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PERFORMAMCE-BASED  LOGISTICS- 

BARRIERS  AND  ENABLERS  TO  EFFECTIVE  IMPLEMENTATION 

Dti  Hank  I  DeVries 

The  Department  of  Defense  is  implementing  Perfoimance-Based  Logistics 
(PBL)  in  both  new  acquisition  programs  and  legacy  programs.  Each  of 
the  Services  is  executing  PBL  policy  at  the  system,  subsystem,  and 
component  level.  The  Services  are  also  working  in  conjunction  with  each 
other  to  implement  PBL  on  joint  programs.  This  study  identifies  the  most 
common  barriers  and  enablers  as  the  Services  go  forward  with  PBL 
implementation,  determines  if  there  are  relationships  between  these  bamers 
and  enablers,  and  also  evaluates  the  success  of  PBL  implementation. 


DEFINING  AND  IMPLEMENTING  PERFORMANCE-BASED 
LOGISTIG  IN  GOVERNMENT 

Dr.DmdBeri[owit^DrJarittderN.D.$upf^,  Dr.JameslSimpsonrattd 
Joan  A.  DkWilliam 


Performance-Based  Logistics  (PBL)  is  a  mechanism  to  integrate  the 
acquisition  and  sustainment  of  various  systems  in  the  Department  of 
Defense.  In  this  article,  we  report  the  results  of  a  research  study  aimed  at 
developing  a  working  definition  of  PBL,  the  drivers  for  its  use,  and  the 
infrastructure  changes  needed  for  its  successful  deployment.  Utilizing  our 
research  findings  and  those  of  previous  related  studies,  we  suggest 
guidelines  for  successfully  implementing  PBL  in  organizations.  We 
conclude  the  article  by  suggesting  some  useful  directions  for  future  research 
to  fully  realize  the  benefits  of  PBL. 


A  ^  A  A  PERFORMANCE-BASED  TECHNOIOGY  ASSESSMENT 
METH0D0106Y  TO  SUPPORT  DOD  ACQUISITION 

Ar  Sherry  Mahaka,  Dr.  hwlJ.  CompenaHon,  mid  Dr,  DenaUD.  Tlppeff 

Many  weapon  system  failures  are  attributed  to  premature  transfer  of 
technology  to  operational  systems.  Insufficient  measures  of  assessing 
technology  readiness  are  major  contributors  to  such  failures.  This  paper 
presents  a  methodology  to  measure  the  performance  risk  of  technology  in 
order  to  determine  its  transition  readiness.  This  methodology  is  referred 
to  as  Technology  Performance  Risk  Index  (TPRI).  The  TPM  can  track 
technology  readiness  through  a  life  cycle,  or  it  can  be  used  at  a  specific 
time  to  support  a  particular  system  milestone  decision.  The  TPRI  is  computed 
using  the  performance  requirements,  the  Degree  of  Difficulty  (DD),  and 
the  unmet  performance.  These  components  are  combined  in  a  closed-l(X)p 
feedback  manner  to  analytically  calculate  the  performance  risk.  The  TPRJ  is 
illustrated  by  an  example  using  published  system  requirements  data. 

SERVICE  LEVEL  AGREEMENTS  AS  VEHICLES  FOR  MANAGING 
ACQUISITION  OF  SOFTWARE-INTENSIVE  SYSTEMS 

€DD  Leonard  T.  Gaines,  DSN  and  Dr  Janies  DretNIkhael 

Service  level  agreements  (SLAs)  can  be  used  as  a  means  to  manage  the 
acquisition  of  software-intensive  systems.  The  SLAs  support  performance- 
based  acquisition  by  stating  in  measurable  terms  the  service  to  be 
performed,  the  level  of  service  that  is  acceptable,  the  way  in  vMch  the 
service  level  is  to  be  measured,  and  the  incentives  for  the  provider  of 
information  technology  products  and  services  to  meet  the  agreed-to  target 
levels  of  quality.  The  SLAs  are  traditionally  used  in  outsourcing  contracts 
for  post-production  support.  This  article  proposes  a  new  approach  by 
using  SLAs  in  software  acquisition  to  support  quality  and  process  control 
throughout  the  entire  lifecycle  of  a  software-intensive  system.  This  article 
defines  SLAs,  discusses  software  quality,  and  describes  how  SLAs  can  be 
utilized  to  incorporate  requirements  pertaining  to  product,  process,  project, 
and  deployment  quality  throughout  the  software  lifecycle. 


LESSONS  LEARNED 

public  and  privah  partnerships  m  support  of 

^%DSD  KRFORMANCE-BASED  LOGISnCS  INITIATIVES— LBSONS 

LEARNED  FROM  DEFENSE  LOGISIK  AGENCY  PARTNERSHIPS 

Dr.  Glenn  LSlarks 

Per  Product  Support  for  the  21st  Century:  A  Program  Manager's  Guide 
to  Buying  Performance,  November,  2001  published  by  the  Defense 
Acquisition  University,  Performance-Based  Logistics  (PBL)  is  the  preferred 
approach  for  implementing  weapon  system  support.  While  PBL  initiatives 
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enhance  weapon  system  support  by  employing  best  commercial  practices 
in  providing  an  integrated  and  performance-based  suppfy  chain,  they 
often  do  not  combine  the  best  of  government  support  with  commercial 
support.  This  results  in  dual  infrastructures,  increased  costs,  and  other 
disadvantages.  This  article  addresses  the  advantages  of  combining  public 
and  private  support  by  discussing  lessons  learned  from  two  PBLs  where 
the  Defense  Logistics  Agency  has  become  the  source  of  supply  to 
commercial  vendors  awarded  PBL  contracts  by  the  Military  Services. 


TUTORIAL 

PERFORMANCE-BASED  SERVICE  ACQUISITION  (PBSA), 

A-76  AND  PERSONAL  SERVICES-Jl  CAUTIONARY  NOTE 

tdward  Allen  Friar 

The  concurrent  emphasis  on  acquiring  services  using  Pterformance-Based 
Service  Acquisition  (PBSA)  and  the  new  A-76  competitive  sourcing 
procedures  gives  rise  to  some  potentially  conflicting  goals  that  acquisition 
personnel  need  to  be  aware  of  in  order  to  avoid  personal  service  contracts. 
A  contract  for  services  can  become  a  personal  services  contract  either  by 
the  w^  it  is  written  or  by  the  way  it  is  administered,  tut  proper  training 
and  planning  can  help  avoid  this  pitfall.  Acquisition  and  contracting 
personnel  need  to  be  informed  about  what  constitutes  personal  services 
and  a>\ure  of  this  limitation  as  it  applies  to  managing  PBSA  contracts. 
This  article  seeks  to  further  define  personal  services  and  offers  some 
suggestions  for  consideration  when  writing  a  Performance  Work  Statement 
(PWS)  or  Statement  of  Objectives  (SOO)  for  a  PBSA. 


CONTRAa  ADMINISTRATION  IN  A  PERFORMANCE-BASED 
ACQUISITION  ENVIRONMENT  IS  SERIOUS  BUSINESS 

John€a¥adias 

Many  of  us  are  eager  to  procure  Performance-Based  Services  Acquisitions 
(PBSA)  as  we  attempt  to  comply  with  procurement  and  acquisition  reform. 
Although  there  is  an  abundance  of  written  material  instructing  us  on  how 
to  develop  and  award  a  PBSA,  we  find  far  less  guidance  on  the  emerging 
realities  in  administering  an  awarded  PBSA.  Contract  administration  in  a 
PBSA  environment  is  mission-critical,  not  to  be  treated  as  an  ancillary 
responsibility  subordinate  to  originating  acquisitions.  This  article 
approaches  this  viewpoint  by  examining  the  post-award  management  of  a 
single  service  in  one  commercial  industry  and  compares  it  to  government 
contracting  practices  particularly  with  an  emphasis  on  legacy  cradle-to- 
grave  organizational  structures  while  also  exploring  the  need  for  a  shift 
in  government  perspective  and  change  in  organizational  practices. 


gmgmmw  nsrANDEVAuiAnoN 

IN  ADYNAMIC  AOHIiSniM  ENVUtONMENT 

(kegoryLBamefle 


Acquisition  reform  and  the  implementation  of  agile  acquisition 
processes  within  the  Air  Force  are  allowing  acquisition  professionals 
greater  flexibility  in  meeting  user  requirements.  A  strong  emphasis  is 
being  placed  on  using  mature  systems  and  technologies,  allowing  new 
programs  to  be  initiated  at  any  point  in  the  acquisition  process 
continuum.  Changes  have  been  incorporated  into  the  test  and 
evaluation  (T&E)  process  to  support  the  increased  flexibility.  Judicious 
use  of  the  various  types  of  recognized  tests  allow  the  program  manager 
(PM)  to  reduce  risk  and  ensure  performance  expectations  are  met.  The 
following  provides  an  overview  of  the  test  tools  available  to  the 
acquisition  professional  and  highlights  the  evolutionary  changes  that 
were  recently  incorporated  into  Air  Force  test  guidance. 
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DEFENSE  ARJ 
EXECUTIVE  EDITOR 

First  I  would  like  to  thank  every  one  of  our  read¬ 
ers  for  the  overwhelming  response  to  our  first 
theme  edition  published  in  August  of  this  year. 

Due  to  the  tremendous  feedback,  we  feel  we  are  on 
the  right  track  in  providing  more  focused  editions 
and  are  looking  forward  to  continuing  to  produce 
theme  editions  in  the  future.  Currently  planned  are 
publications  focused  on  Communities  of  Practice  (CoP)  and  system-of-systems  acqui¬ 
sition.  Anticipated  future  editions  will  feature  articles  on  Transformation  and  Leader¬ 
ship,  Systems  Engineering  Best  and  Worst  Practices,  and  Technology  Transition  and 
Implications.  If  you  are  doing  research  in  any  of  these  areas  and  would  like  to  submit 
an  article,  please  contact  Norene  Taylor,  managing  editor,  Defense  ARJ,  at 
norene.taylor@dau.mil.  Similarly,  if  you  are  interested  in  being  an  article  reviewer  in 
any  of  these  areas,  we  would  like  to  hear  from  you.  We  welcome  your  participation 
in  helping  us  to  ensure  this  journal  meets  the  needs  of  the  Acquisition,  Technology, 
and  Logistics  (AT&L)  workforce. 

The  thrust  of  this  edition  is  Performance-Based  Acquisition,  and  as  many  of  you 
know,  the  emphasis  on  performance  within  the  government  began  in  1992  with  the 
passage  of  the  Government  Performance  and  Results  Act.  It  was  further  emphasized 
by  former  Vice  President  A1  Gore  in  his  National  Performance  Review  and  also  in 
subsequent  Hammer  Award  presentaions.  Today  of  course,  we  accept  performance  as 
a  focus  for  all  acquisition  disciplines  and  processes.  Consequently,  the  articles  pre¬ 
sented  in  this  edition,  touch  on  various  areas  of  the  acquisition  process  and  ways  in 
which  performance  is  affecting  the  things  we  do  and  the  way  we  think 

Our  featured  author.  Dr.  Hank  DeVries,  is  an  instructor  at  the  Defense  Acquisition 
University  (DAU)  West  Region  and  is  interested  in  ways  in  which  Performance-Based 
Logistics  (PBL)  is  being  implemented.  His  research  article  entitled,  “Performance-Based 
Logistics-Barriers  and  Enablers  to  Effective  Implementation,”  considers  the  practical 
aspects  of  PBL  and  identifies  those  items  within  this  practice  that  can  lead  to  a  suc¬ 
cessful  endeavor  and  those  barriers  that  can  confound  and  seriously  effect  its  success¬ 
ful  implementation. 

The  following  three  articles  document  other  applied  research  activities.  Berkowitz, 
Gupta,  Simpson,  and  McWilliams  in  their  paper,  “Defining  and  Implementing  Perfor¬ 
mance-Based  Logistics  in  Government,”  provide  a  working  definition  of  PBL  and 
address  the  infrastructure  changes  needed  for  its  successful  deployment.  Mahafza, 
Componation,  and  Tippett’s,  “A  Performance-Based  Technology  Assessment  Method¬ 
ology  to  Support  DoD  Acquisition/'  focuses  on  the  topic  of  performance  risk  and 


suggests  the  use  of  a  Technology  Performance  Risk  Index  (jpRi)  as  a  means  of 
tracking  technology  readiness.  Gaines  and  Michael’s  article  entitled,  'Managmg  Ac¬ 
quisition  of  Software-Intensive  Systems  with  Service  Level  Agreements,”  provide  a 
prescriptive  approach  to  managing  in  a  performance-based  environment. 

The  next  article  focuses  on  lessons  learned  to  assist  future  performance-based 
acquisition  efforts.  Starks  paper  entitled,  “Public  and  Private  Partnerships  in  Support¬ 
ing  Performance-Based  Logistics  Initiatives/’  focuses  on  lessons  learned  from  com¬ 
bining  the  best  of  government  support  with  commercial  support  for  providing  an 
int^rated,  performance-based  supply  chain. 

Our  last  three  articles  are  tutorials  regarding  the  implementation  of  performance- 
based  efforts.  Friar’s,  “Performance-Based  Service  Acquisition  (PBSA),  A-76  and 
Personal  Services,”  discusses  the  issue  of  personal  services  in  a  performance-based 
arena.  Cavadias’  paper  entitled,  “Contract  Administration  in  a  Performance-Based 
Acquisition  Environment  is  Serious  Business,”  looks  at  ways  in  which  to  conduct 
contract  administration  in  a  performance-based  environment.  Finalfy,  Barnette’s  “Test 
and  Evaluation  in  a  Dynamic  Acquisition  Environment,”  talks  about  the  incorpora¬ 
tion  of  new  test  and  evaluation  procedures  to  reduce  risk  and  ensure  performance 
expectations. 

Therefore,  we  have  provided  you  with  a  wide  range  of  articles  on  the  topic  of 
performance-based  acquisition.  As  a  follow-on  activity  to  our  Journal,  we  will  be 
conducting  a  discussion  forum  in  January  based  on  this  edition  using  the  authors 
as  discussion  leads.  Please  look  for  our  advertisements  toward  the  end  of  this  jour¬ 
nal.  We  hope  that  you  are  interested  enough  in  the  topic  of  Performance-Based 
Acquisition  to  join  us  or  to  provide  questions  for  discussion.  We  intend  to  use  the 
Acquisition  Research  Community  of  Practice  interest  area  on  the  Acquisition  Com¬ 
munity  Connection  as  a  vehicle  for  sharing  discussion  information. 


Dr.  Beryl  A.  Harman 
Executive  Editor 
Defense  ARJ 
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PERFORMANCE-BASED 
LOGISTICS- 
BARRIERS  AND  ENABLERS 
TO  EFFECTIVE 
IMPLEMENTATION 

DR.  HANK  I  DEVRIES 


The  Department  of  Defense  is  implementing  Performance-Based  Logistics 
(PBL)  in  both  new  acquisition  programs  and  legacy  programs.  Each  of  the 
Sen/ices  is  executing  PBL  policy  at  the  system,  subsystem,  and  component 
level.  The  Services  are  also  working  in  conjunction  with  each  other  to 
implement  PBL  on  joint  programs.  This  study  identities  the  most  common 
barriers  and  enablers  as  the  Services  go  forward  with  PBL  implementation, 
determines  if  there  are  relationships  between  these  barriers  and  enablers, 
and  also  evaluates  the  success  of  PBL  implementation. 


Traditionally,  support  for  weapon  systems  in  the  Department  of  Defense  (DoD) 
centered  around  ten  or  eleven  logistics  elements,  split  between  acquisition- 
related  activities  at  the  front  end  of  the  life  cycle-  and  sustainment-related 
activities  at  the  back  end.  Metrics  focused  on  the  logistics  elements  themselves  and 
internal  processes  often  having  little  direct  relationship  to  warfighter  require¬ 
ments.  The  shift  toward  Integrated  Logistics  Support  attempted  to  wrap  together 
the  distinct  logistics  elements  into  a  coordinated  approach,  but  there  was  still  the 
disjointed  acquisition  versus  sustainment-support  issue  and  the  lack  of  a  linkage 
between  supportability  measures  and  warfighter  needs.  In  addition,  choice  of 
support  providers  was  often  an  all  or  nothing  proposition;  either  entirely  organic 
(DoD)  or  entirely  commercial  (CLS  or  contractor  logistics  support).  The  advent 
of  Total  Life  Cycle  Systems  Management  (TLCSM)  and  Pferformance-based  Lo¬ 
gistics  (PBL)  addressed  all  of  these  issues. 
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The  TLCSM  mandated  a  new  focus  by  program  managers  toward  the  entire  life 
cycle,  firmly  linking  acquisition  and  sustainment  activities  into  an  integrated  pro¬ 
cess.  To  measure  success,  PBL  required  that  supportability  metrics  be  directly  related 
to  performance  outcomes  for  the  warfighter,  and  PBL  also  offered  a  choice  of 
organic  and  commercial  support  providers  for  picking  the  right  combination  in 
achieving  best  value  to  the  program.  A  succinct  definition  quoted  from  a  recent 
report  by  the  Center  for  the  Management  of  Science  and  Technology  at  the  Uni¬ 
versity  of  Alabama  in  Huntsville  defines  PBL  as,  “an  integrated  acquisition  and 
sustainment  strategy  for  enhancing  weapon  system  capability  and  readiness,  vdiere 
the  contractual  mechanisms  will  include  long-term  relationships  and  appropriately 
structured  incentives  with  service  providers,  both  organic  and  non-organic,  to  support 
the  end  user’s  (warfighter’s)  objectives”  (Berkowitz,  et  al.,  2003,  p.  5). 


The  TICSJUI  mandated  a  new  fo€us  by  program 
managers  toward  the  entire  life  tyde,  firmly 
linking  atquisition  and  sustainment  attivities 
into  an  integrated  protess. 


Implementation  of  PBL  was  mandated  in  September,  2001  in  the  Quadrennial 
Defense  Review  (QDR),  and  initial  guidance  was  promulgated  by  the  Office  of 
Secretary  of  Defense  (OSD)  (Aldridge,  2002).  The  OSD  issued  a  Product  Support 
Guide  that  provided  a  strat^y  for  executing  PBL  (Morales,  2001).  Subsequently,  each 
of  the  Services  provided  implementation  guidance  to  their  programs  (Bolton,  2002; 
Schneider,  2002).  In  accordance  with  the  FY03  Defense  Policy  Guidance,  the  scope 
of  the  programs  to  be  considered  for  PBL  implementation  included  all  new  weapon 
systems  and  all  Acquisition  Category  (ACAT)  I  and  II  fielded  systems  (Y)ung,  2003). 
The  importance  of  sustainment  in  the  program  life  cycle  and  in  implementing  PBL 
was  recognized.  To  ensure  the  requisite  priority  on  sustainment  issues  within  program 
offices  and  to  ease  the  PBL  implementation  efforts,  the  concept  of  TLCSM  was 
promulgated  (Aldridge,  2003). 

Total  Life  Cycle  Systems  Management  emphasizes  an  early  fpcus  on  sustain¬ 
ment  in  the  program  management  office,  making  the  program  manager  respon¬ 
sible  for  all  activities  associated  with  the  acquisition,  development,  production, 
fielding,  sustainment,  and  disposal  of  a  weapon  system  across  its  life  cycle.  This 
was  a  significant  paradigm  shift  from  traditional  program  management  focus  on 
the  early  phases  (acquisition,  development,  fielding)  of  the  life  cycle.  To  support 
the  decision-making  process  in  selecting  organic  and  commercial  support  pro¬ 
viders,  OSD  promulgated  guidelines  for  conducting  a  Business  Case  Analysis 
(BCA)  (Wynne,  2004a).  In  addressing  the  performance  metrics’  relationship  to 
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desired  outcomes,  OSD  provided  some  common  examples  such  as  operational 
availability  and  logistics  footprint  (Wynne,  2004b). 

The  Services  began  encountering  problems  in  implementing  PEL,  both  for 
new  and  existing  programs.  There  were  existing  cultural  and  structural  barriers 
that  inhibited  effective  implementation.  On  the  other  hand,  there  were  a  number 
of  enablers  that  were  being  utilized  for  more  effective  implementation.  These 
barriers  and  enablers  were  the  subject  of  numerous  program  briefings  and  re¬ 
ports  presented  at  a  number  of  conferences  and  road  shows  over  the  last  couple 
of  years. 

This  research  study  intends  to  identify  the  most  frequently  impacting  barriers  and 
enablers  to  effective  PEL  implementation  within  DoD,  how  th^  influence  PEL 
implementation,  and  recommend  strategy/actions  that  will  facilitate  more  effective 
implementation  for  new  and  legacy  programs. 

LITERATURE  REVIEW 

An  extensive  search  was  conducted  on  the  Internet  to  identify  current  PEL  policy 
and  implementation  guidelines.  A  review  of  OSD  and  Service  Web  sites,  as  well  as 
some  industry  Web  sites,  was  completed.  Eriefings  from  a  number  of  conferences 
were  obtained,  showing  the  status  of  several  programs  undergoing  PEL  implementation. 
Also,  there  were  ongoing  discussions  and  correspondence  regarding  PEL  implemen¬ 
tation  and  problems  encountered  with  a  number  of  practitioners  within  the  Services. 
A  review  was  conducted  of  existing  DAU  curriculum  in  Performance-Eased 
Acquisition  and  Performance-Eased  Logistics.  Through  participation  in  PEL  con¬ 
ferences  and  road  shows,  there  were  discussions  with  key  policymakers  and 
implementers.  Eased  on  the  preliminary  literature  survey  and  feedback  from 
practitioners,  it  was  apparent  that  there  were  numerous  instances  of  misunder¬ 
standing  of  the  PEL  concept,  resistance  to  its  initiatives,  and  problems  in  imple¬ 
mentation. 


RESEARCH  QUESTIONS  AND  HYPOTHESES 

The  following  questions  were  posed  to  frame  the  research  effort: 

1.  What  are  the  barriers  and  how  do  they  influence  PEL  implementation? 

2.  What  are  the  enablers  and  how  do  they  influence  PEL  implementation? 

3.  What  strategy/actions  would  lead  to  more  successful  PEL  implementation? 

In  reference  to  question  1,  seven  key  barriers  were  identified  through  the  efforts 
of  the  Literature  Review.  These  barriers  were: 
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1.  Funding  restrictions/inflexibility  (e.g.,  Working  Capital  Fund,  various  appro¬ 
priations/transfer  and  expiring  funds  rules,  limited  Program  Manager  [PM] 
control  over  Operation  and  Maintenance  [O&M]). 

2.  Statutory/regulatory  requirements  (e.g.,  Title  10,  service  policies). 

3.  Old  paradigms/culture  (e.g.,  organic  versus  commercial,  parts  management  versus 
performance  management,  minimize  contractors  on  the  battlefield). 

4.  Existing  infrastructure/bureaucracy  (e.g.,  PM  office  structure,  stovepiping,  short 
PM  tours). 

5.  Technical  data  rights  issues. 

6.  Lack  of  PEL  awareness/training. 

7.  Inability  to  incentivize  organic  providers. 

In  reference  to  question  2,  seven  key  enablers  were  identified  through  the  efforts 

of  the  Literature  Review.  These  were: 

1.  Supply  Chain  Management  (e.g.,  end-to-end  customer  support,  enterprise 
integration). 

2.  Strategic  alliances/partnerships  (e.g.,  depot  partnering,  joint  ventures). 

3.  Performance-based  contracting  (e.g.,  incentivizing  performance). 

4.  Performance-based  metrics. 

5.  Total  Life  Cycle  Systems  Management  (TLCSM)  perspective. 

6.  Adoption  of  Commercial  Off-the-Shelf  (COTS)/Best  Commercial  Practices. 

7.  Reduction  in  Total  Ownership  Cost  (RTOC)  initiative. 

Based  on  the  research  questions,  six  hypotheses  were  developed: 

1 .  There'  is  an  indirect  relationship  between  the  number  of  barriers  and  the  success 
of  PEL  implementation. 

2.  There  is  a  direct  relationship  between  the  mitigation  of  barriers  and  the  success 
of  PEL  implementation. 

3 .  There  is  an  indirect  relationship  between  the  influence  of  barriers  (after  mitigiation) 
and  the  success  of  PEL  implementation. 


Success  of  PBL 
Implementation 


LEGEND 

Independent  variable 
Moderating  variable 
Dependent  variable 


PBL  Performance-Based  Logistics 


FIGURE  1.  RESEARCH  MODEL 


4.  There  is  a  direct  relationship  between  the  number  of  enablers  and  the  success 
of  PBL  implementation. 

5.  There  is  a  direct  relationship  between  the  enhancement  of  enablers  and  the  success 
of  PBL  implementation. 

6.  There  is  a  direct  relationship  between  the  influence  of  enablers  (after  enhancement) 
and  the  success  of  PBL  implementation. 

The  research  model  in  Figure  1  graphically  displ^s  these  hypotheses  and  associated 
variables. 


RESEARCH  METHODOLOGY 

This  research  study  was  primarily  qualitative  in  the  measurement  of  variables. 
Correlational  research  was  conducted  using  surveys  to  obtain  primary  data.  Surveys 
were  selected  as  an  effective  method  to  obtain  data  from  program  offices  \)vhere  they 
are  implementing  PBL  in  their  programs.  The  statistical  test  used  for  all  six 
correlation  Itypotheses  was  the  Pearson  product-moment  (at  least  one  of  two  variables 
in  each  hypothesis  is  ratio  or  interval  type  data).  Due  to  the  small  size  of  the  sample 
and  the  fact  that  the  dependent  variable  consisted  of  ordinal  type  date,  the  Spearman 
rank-order  r  test  was  also  conducted  on  all  six  hypotheses.  Results  were  then  com- 


247 


pared  with  the  results  of  the  Pearson  product-moment  test.  No  significant  differences 
were  noted. 


RESEARCH  SURVEY 

A  data  survey  was  created  on  the  Web  and  instructions  sent  out  to  key  PEL 
points  of  contact  (POC)  within  each  of  the  Services.  The  Service  POC’s  instructed 
program  managers  that  had  undergone  PEL  implementation  within  their  respective 
Service  to  fill  out  the  data  suiv^.  There  were  a  total  of  26  program  managers  that 
responded  to  the  survey.  Of  the  26  programs,  10  were  Joint,  9  were  Army,  and  7 
were  Navy/Marine  Corps.  No  Air  Force  specific  programs  responded.  Eoth  new 
and  legacy  programs  participated.  Of  the  26  programs,  11  were  new  and  15  were 
legacy.  Another  distinguishing  factor  was  the  scope  of  the  PEL;  implemented  at 
the  system,  subsystem,  or  component  level.  Of  the  26  programs,  11  were  system 
level,  9  at  subsystem  level,  4  at  component  level,  and  2  did  not  distinguish.  A  final 
distinguishing  factor  was  the  impact  of  PEL  on  logistics  elements.  Three  primaiy 
logistics  elements  were  chosen:  supply,  maintenance,  and  transportation.  Of  the  26 
programs,  13  impacted  all  three  elements,  4  impacted  supply  and  maintenance,  1 
impacted  supply  and  transportation,  2  impacted  only  supply,  3  impacted  only 
maintenance,  1  impacted  only  transportation,  and  2  did  not  distinguish.  Figure  2 
provides  a  summary  chart. 


RESEARCH  FINDINGS 


O) 

E 

E 

O) 

o 
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FIGURE  a.  PROGRAM  SUAAAAARY  CHART 


Of  the  26  programs  surveyed,  17  identified  funding  as  a  barrier;  13  identified 
statutory/re  gulatoiy,  culture,  and  lack  of  PBL  training  as  barriers;  12  identified  existing 
infrastructure  as  a  barrier;  11  identified  technical  data  rights  issues  as  a  barrier;  and 


FIGURE  3.  BARRIERS  IDENTIFIED 


FIGURE  4.  ENABLERS  IDENTIFIED 
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6  identified  lack  of  organic  depot  incentives  as  a  barrier.  For  the  same  26  programs 
surv^ed,  16  identified  performance  metrics  as  an  enabler;  14  identified  performance- 
based  contracting,  TLCSM,  and  COTS/best  commercial  practices  as  enablers;  11  iden¬ 
tified  partnering  as  an  enabler;  and  8  identified  supply  chain  management  and  RTOC 
as  enablers.  Summary  charts  of  the  results  are  shown  in  Figures  3  and  4. 

Of  the  6  hypotheses  tested,  the  data  analysis  supported  3  hypotheses  at  the 
0.10  significance  level  using  both  the  Pearson  product-moment  test  and  the 
Spearman  rank- order  r  test.  Of  the  3  supported  hypotheses,  2  were  supported  at  the 
0,05  significance  level  using  the  Pearson  product-moment  test  and  1  was  supported 
at  the  0.05  significance  level  using  the  Speannan  rank-order  r  test.  The  3  supported 
hypotheses  were: 

1 .  There  is  a  direct  relationship  between  the  number  of  enablers  and  the  success  of 
PBL  implementation. 

2.  There  is  a  direct  relationship  between  the  enhancement  of  enablers  and  the  success 
of  PBL  implementation. 

3 .  There  is  a  direct  relationship  between  the  influence  of  enablers  (after  enhancement) 
and  the  success  of  PBL  implementation. 


TABLE  1. 

RESULTS  SUMAAARY 


Hypothesis 

Pearson 

Correlation 

Coefficient 

Significance 

Level 

(Pearson) 

Spearman 

Correlation 

Coefficient 

Significance 

Level 

(Spearman) 

Indirect  relationship  between 
number  of  barriers  and  success  of 
Performance-Based  Logistics  (PBL). 

-0.102 

0.68 

-0.154 

0.53 

Direct  relationship  between 
mitigation  of  barriers  and  success 
of  PBL. 

-0.162 

0.51 

-0.130 

0.60 

Indirect  relationship  between 
influence  of  barriers  (after  mitigation) 
and  success  of  PBL. 

-0.192 

0.43 

-0.227 

0.35 

Direct  relationship  between 
number  of  enablers  and  success  of 
PBL. 

0.414 

0.08 

-0.443 

0.06 

Direct  relationship  between 
enhancement  of  enablers  and 
success  of  PBL. 

0.540 

0.02 

0.456 

0.05 

Direct  relationship  between 
influence  of  enablers  (after 
enhancement)  and  success  of  PBL. 

0.533 

0.02 

0.393 

0.10 
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Of  the  6  hypotheses  tested,  the  data  analysis  did  not  provide  sufficient  support  for 

3  of  the  hypotheses.  They  are  as  follows: 

1 .  There  is  an  indirect  relationship  between  the  number  of  barriers  and  the  success 
of  PBL  implementation. 

2.  There  is  a  direct  relationship  between  the  mitigation  of  barriers  and  the  success 
of  PBL  implementation. 

3 .  There  is  an  indirect  relationship  between  the  influence  of  barriers  (after  mitigation) 
and  the  success  of  PBL  implementation. 

A  summary  of  the  results  is  shown  in  Table  1. 


CONCLUSIONS 

Based  on  the  results  of  the  survey  and  data  analysis,  there  appears  to  be  some 
relationship  between  the  identified  enablers  and  the  success  of  PBL  implementation. 
The  most  frequent  enablers  that  appeared  to  influence  success  were  performance 
metrics,  performance-based  contracting,  TLCSM,  and  COTS/Best  Commercial 
Practices.  This  is  in  alignment  with  the  level  of  emphasis  in  these  areas  from  both 
a  policy  and  training  perspective  within  DoD.  Those  enablers  influencing  fewer 
programs  were  supply  chain  management  and  RTOC.  Although  eertainly  important 
from  a  broad  PBL  perspective,  it  may  be  more  challenging  for  respondents  to  link 
these  concepts  to  PBL  execution  at  the  program  office  level. 

As  in  most  research  studies,  all  the  hypotheses  may  not  be  supported  fiom  the  data 
analysis.  In  this  case,  the  hypotheses  dealing  with  banners  to  PBL  implementation  and 
their  influence  on  success  were  not  supported  at  the  requisite  sigmficance  level.  This 
may  be  due  to  the  small  sample  size  (26)  and/or  the  inability  to  understand  the  true 
impacts  of  barriers  on  PBL  execution.  It  was  apparent  in  the  literature  survey  that  a 
number  of  activities  view  barriers  as  a  significant  issue  in  their  implementation  efforts 
and  that  policymakers  are  coming  out  with  initiatives  to  mitigate  some  of  those  bar¬ 
riers.  What  the  research  study  did  show  was  that  funding  seems  to  be  the  most  fre¬ 
quently  encountered  barrier  followed  by  statutory/regulatory,  culture,  and  lack  of  PBL 
training.  The  least  encountered  barrier  was  lack  of  organic  depot  incentives,  which 
may  be  partly  due  to  the  use  of  commercial  depots  by  some  of  the  programs  surveyed. 


SUMMARY/RECOMMENDATIONS 

Based  on  the  research  study  findings,  policymakers  in  DoD  should  continue  to 
focus  on  initiatives  that  eneourage  the  use  of  enablers  such  as  performance  metrics, 
performance-based  contracting,  and  use  of  COTS/Best  Commercial  Practices.  They 
should  look  at  w^s  to  more  closely  link  concepts  such  as  Supply  Chain  Management 
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and  RTOC  to  program  execution  so  that  implementors  of  PEL  realize  the  practical 
application  of  those  concepts.  Policymakers  should  increase  their  efforts  to  mitigate 
barriers  in  the  funding,  statutory/regulatory,  and  training  areas.  Replacement  of  the 
Planning,  Programming,  and  Budgeting  System  (PPBS)  with  Planning,  Programming, 
Budgeting,  and  Execution  (PPBE)  and  relaxation  of  regulatory  requirements  (DoD 
5000  series/Defense  Acquisition  Guidebook)  are  starting  to  have  some  impact,  along 
with  a  new  focus  on  Performance-Based  Acquisition  training  at  the  Defense  Acqui¬ 
sition  University  through  classroom,  online,  and  continuous  learning  activities.  These 
efforts  need  to  continue  and  be  reinforced  by  service  policy  and  training  efforts. 

At  the  program  office  level,  logisticians  need  to  work  in  close  concert  with  the 
program  manager  and  other  acquisition  disciplines  to  address  performance  issues  and 
ensure  metrics  are  linked  closely  with  warfighter  outcomes.  Contracting  officers  need 
to  work  closely  with  logisticians  when  drafting  contracting  strategy  and  building  in¬ 
centives  into  contracts.  Financial  managers  and  logisticians  need  to  jointly  develop  life 
cycle  cost  estimates  and  come  up  with  innovative  approaches  within  the  funding 
constraints  and  statutory  guidelines  to  reduce  total  ownership  cost.  Logisticians  need 
to  develop  objective  business  case  analyses  to  support  smart  decisions  on  the  right  mix 
of  support  providers  to  optimize  warfighter  performance  outcomes. 

In  summary,  PBL  along  with  TLCSM  have  required  a  paradigm  shift  in  how  we 
view  program  life  cycles  and  supportability.  There  are  a  lot  of  challenges  or  barriers 
that  inhibit  our  ability  to  be  effective.  There  are  also  a  lot  of  enablers  that  increase  our 
ability  to  be  successful  in  implementing  PBL.  If  policymakers  working  in  concert  with 
program  offices  can  continue  to  mitigate  the  barriers  and  enhance  the  enablers,  we  can 
offer  a  better  product  to  the  warfighter  that  will  meet  or  exceed  their  performance 
requirements  while  providing  long  term  savings  to  the  program.  Only  in  this  way  can 
we  both  meet  the  increasing  challenges  of  the  new  threat  environment  and  stay  within 
the  tightening  budget  constraints  of  today  and  in  the  future. 


Dr.  Hank  DeVries  is  currently  serving  as  the  Associate  Dean  for 
Outreach  and  Performance  Support  at  the  Defense  Acquisition 
University,  West  Region,  San  Diego,  CA.  He  is  a  Certified  Professional 
Logistician  and  has  more  than  26  years  experience  in  acquisition  and 
sustainment  logistics.  He  earned  a  master  of  science  degree  in  material 
logistics  support  management  from  the  Nava!  Postgraduate  School, 
and  a  doctor  of  business  administration  degree  in  strategic 
management  from  Alliant  International  University.  Dr.  DeVries  is  a 
member  of  the  Defense  Acquisition  Corps  and  is  certified  at  DAWIA 
Level  ill  in  Acquisition  Logistics  and  Program  Management.  In  addition 
to  his  work  as  the  Associate  Dean,  he  also  teaches  acquisition 
management,  logistics,  and  program  management  courses  for  the 
Defense  Acquisition  University. 

{E-mail  address:  hank. devries@dau. mil) 
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Rerformance-Based  Logistics  (PBL)  is  a  mechanism  to  integrate  the  acquisition 
and  sustainment  of  various  systems  in  the  Department  of  Defense.  In  this 
article;  we  report  the  results  of  a  research  study  aimed  at  developing  a 
working  definition  of  PBL,  the  drivers  for  its  use,  and  the  infrastructure 
changes  needed  for  its  successful  deployment.  Utilizing  our  research  find¬ 
ings  and  those  of  previous  related  studies,  we  suggest  guidelines  for  suc¬ 
cessfully  implementing  PBL  in  organizations.  We  conclude  the  article  by 
suggesting  some  useful  directions  for  future  research  to  fully  realize  the 
benefits  of  PBL. 


Product  acquisition  and  sustainment  have  traditionally  been  separate  and  not 
necessarily  equal  concerns.  The  government’s  primary  focus  has  been  on  the 
acquisition  of  technology  and  systems.  Additionally,  the  government  has  had 
a  number  of  secondary  concerns:  sustainment  of  the  system,  technology  transfer,  and 
the  development  of  an  industrial  base  to  support  the  system  long  term.  The  environ¬ 
ment  for  government  acquisition  creates  consequences  for  major  programs  that  span 
years,  if  not  decades.  As  the  government  strives  to  understand  how  to  generate  the 
best  value  for  its  systems,  it  is  appropriate  to  study  experiences  in  the  Department 
of  Defense  (DoD)  and  in  Industry  in  order  to  maximize  performance  for  the  life  of 
the  system. 
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The  ultimate  goal  in  an  acquisition  strategy  is  to  build  both  partnerships  and 
relationships  that  align  the  goals  of  all  organizations  for  the  duration  of  the  program. 
Once  the  competition  for  the  initial  acquisition  of  a  system  has  occurred,  the  ability 
of  the  government  and  the  contractor  to  make  substantial  changes  in  the  system  is 
typically  limited.  Since  some  acquisition  efforts  last  for  decades,  it  is  essential  for  the 
parties  to  explore  the  acquisition  strategy  carefully  before  embarking  on  a  course  of 
action.  This  is  especially  so  as,  over  the  life  cycle  of  most  systems,  it  has  been 
estimated  that  about  30  percent  of  all  dollars  spent  are  used  to  acquire  the  system, 
while  the  remaining  70  percent  of  all  dollars  are  used  for  support. 

The  goal  of  both  acquisition  and  sustainment  is  to  gain  the  most  efficient  and 
effective  performance  of  the  system  for  its  entire  life.  In  doing  so,  it  is  important  to 
realize  that  acquisition  and  sustainment  are  not  separate  but  simultaneous  and 
integrative  issues  that  require  analysis  and  synthesis  throughout  the  product  life  cycle. 
Ultimately,  the  challenge  for  the  program  manager  is  to  stmcture  optimal  relationships 
with  contractors  through  the  use  of  appropriate  contractual  mechanisms,  agreements, 
and  incentives. 

The  Department  of  Defense  initiated  a  long-term  program  to  link  performance  to 
acquisition  through  a  concept  called  Performance-Based  Logistics  (PBL),  which 
represents  an  integrated  Perfonnance-Based  Environment  (PBE)  for  both  acquisition 
and  sustauunent.  This  is  very  appropriate  since  dollars  spent  on  maintenance  continue 
to  increase  as  systems  age.  Since  the  inception  of  PBL,  various  agencies  have  tried 
to  develop  definitions,  implementation  guidelines,  and  infrastructure  to  attain  the 
goal  of  acquisition  and  sustainment  integration  through  performance-based  initia¬ 
tives.  While  several  organizations  in  various  branches  of  the  DoD  have  attempted 
to  use  PBL  approaches  in  acquisition  and  sustainment  efforts,  no  clear  and  univer- 
salfy  acceptable  definition  of  PBL  exists.  Therefore,  there  is  no  clear  understanding 
of  the  drivers  of  PBL.  Hence,  implementation  guidelines  for  PBL  are  at  best  ad-hoc 
and  incomplete.  This  situation  undermines  the  DoD’s  ability  to  use  PBL  to  make 
Defense  operations  more  responsive. 

The  purpose  of  this  article  is  to  identify  the  issues  and  complexities  of  the 
relationships  that  exist  in  making  the  transition  to  the  PBL  environment.  Utilizing  the 
relational  exchange  theory  and  new  product  development  literature,  along  with  the 
combined  knowledge  and  resources  of  the  government  and  in  Industry,  we  develop 
a  conceptual  and  vrorking  definition  of  PBL,  identify  the  drivers  for  the  deployment 
of  PBL,  propose  the  needed  infrastructure  changes  to  be  effective  and  efficient  in 
using  PBL,  and  outline  guidelines  found  useful  in  implementing  PBL.  Finally,  we 
conclude  the  paper  by  summarizing  our  findings  and  suggesting  some  directions  for 
future  research  to  successfully  implement  PBL. 


RESEARCH  PROCESS 


In  this  section,  we  briefly  describe  the  combination  of  research  processes  and 
methodologies  used  to  achieve  the  goals  of  this  research.  Basical^,  we  used  interviews 
as  the  primary  vehicle  to  gather  information  about  the  definition  and  deployment  of 
PBL.  In  addition  to  interviews,  we  hosted  roimdtable  discussions.  We  also  participated 
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in  an  Army  Materiel  Command  (AMC)-wide  PBL  video  conference  at  the  Aviation  and 
Missile  Command  (AMCOM).  For  the  interviews,  we  used  an  emeigent  design  process 
that  employs  a  predetennined  set  of  questions  tp  start  the  interview  process.  In  this 
approach,  the  set  of  questions  are  altered  over  time  to  reflect  what  was  learned  in 
previous  interviews.  Before  the  interview  process  began,  each  respondent  was  informed 
that  the  purpose  of  the  research  was  to  develop  a  workable  definition  and  implemen¬ 
tation  approach  for  the  transition  to  a  PBL  environment. 


While  a  DoD-wide  study  of  PBl  efforts  is 
useful,  information  is  also  available  from  the 
Industry's  logistirs  strategies  and  approarhes 
to  solving  problems. 


m 


Following  a  review  of  the  PBL  literature,  we  identified  individuals  and  oiganizations 
engaged  in  PBL-type  activities.  We  then  grouped  potential  respondents  into  four 
categories:  Army,  Navy,  Air  Force,  and  Industry.  We  conducted  in-depth  interviews, 
often  lasting  many  hours,  with  contractors  and  DoD  project  managers.  For  example, 
interviews  were  conducted  at  Warner  Robins  Air  Force  Base,  Wright  Patterson  Air 
Force  Base,  Naval  Inventory  Control  Point  (NAVICP)  in  Philadelphia,  and  at  PBL 
Conferences.  We  used  each  interview  to  document  and  investigate  how  PBL  is  both 
defined  and  operationalized.  In  some  instances,  we  also  conducted  telephone  inter¬ 
views  including  those  with  people  ffotn  Headquarters  (HQ)  Navy,  General  Account¬ 
ing  Office  (GAD),  Defense  Logistics  Agency  (DLA),  RAND,  and  selected  contrac¬ 
tors.  In  general,  there  was  a  very  high  level  of  cooperation  at  all  levels  among  both 
government  and  Industry  participants. 


RESPONDENTS 

A  review  of  literature  on  PBL-related  activities  revealed  that,  in  1998,  DoD  es¬ 
tablished  30  sustainment  pilot  programs,  of  which  24  adopted  some  type  of  inno¬ 
vative  product  support  strategies  (Product  Support  for  the  21st  Century,  2001).  We 
contacted  project  managers  from  tiie  pilots  to  schedule  visits  and  interviews.  Table 
1  lists  the  30  initial  programs  and  highlights  the  programs  interviewed  by  our  re¬ 
search  team. 

In  addition  to  the  pilot  programs  listed  m  Table  1,  we  also  interviewed  managers 
from  the  Soldier  Focused  Logistics  (SFL)  progr^,  a  collaborative  effort  between 
AMCOM  and  the  Cargo  Helicopters  Project  Manager’s  (PM)  Office  to  support  the 
CH-47  fleet  sustainment  using  PBL  strategies. 
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fABLE  1. 

PILOT  PROGRAMS  FOR  PRODUCT  SUPPORT  STRATEGIES 


DoD  PILOTS  FOR  PRODUCT  SUPPORT  STRATEGIES 


Abrams  M-1  Tank 

Advanced  Amphibious 

Assault  Vehicle  (AAAV) 

Airborne  Warning  and 

Control  System  (AWACS) 

Advanced  Field  Artillery  Tactical 

Data  System  (AFATDA) 

AEGIS  Cruiser 

B-1B  Lancer 

Apache  AH-64 

ASE/CASS 

C-17  Globemaster 

Chinook  CH-47 

Common  Ship 

C-5  Galaxy 

Comanche  RAH-66 

CVN-68 

Cheyenne  Mountain  Complex 

Crusader 

EA-6B  Prowler 

F-117  Nighthawk 

Guardrail/Common  Sensor 

H-60  Helicopter 

F-16  Falcon 

Heavy  Expanded  Mobility 
TacticalTrucks(HEMTT) 

Landing  Platform  Dock-17  (LPD-17) 

Joint  Surveillance  Target  Attack 
Radar  System  (J-STARS) 

High  Mobility  Artillery 

Rocket  System  (HIMARS) 

Medium  Tactical  Vehicle 

Replacement  (MTVR) 

KC-135  Stratotanker 

TOW/TTAS 

Standoff  Land  Attack  Missile- 
Expanded  Response  (SLAM-ER) 

Space-Based  Infrared 

Systems  (SBIRS) 

ASE/CASS = Aviation  Support  Equipment  Consolidated  Automated  Support  System 
TOW/ITAS  =  Tube-launched,  Optically-tracked,  Wire-guided  Improved  Target  Acquisition  System 

Highlighted  programs  were  Included  In  our  Performance-Based  Logistics 
_ research  through  Interviews  or  presentations. 

While  a  DoD-wide  study  of  PBL  efforts  is  useful,  information  is  also  available 
from  the  Industry’s  logistics  strategies  and  approaches  to  solving  problems.  There¬ 
fore,  we  interviewed  Industry  managers  from  AutoZone,  UPS,  Target,  Caterpillar, 
Intergraph,  Dell  Computers,  Royal  Caribbean  Cruises,  and  the  University  of  Toronto. 
Since  the  term  Performance-based  Logistics  is  not  used  in  the  private  sector,  we 
widened  the  scope  of  logistics  to  include  inventory  management,  spare  parts 
acquisitions  and  repair,  and  maintenance  activities. 


DEFINITION  OF  PBL 

The  first  objective  of  our  research  was  to  review  existing  definitions  of  PBL  used 
in  the  Army,  Navy,  Air  Force,  and  Industry  to  provide  insight  into  the  tenets  of  PBL. 
We  found  no  single  definition  for  PBL.  Yet,  various  PBL  definitions  revealed  several 
common  themes.  The  three  main  themes  are:  1)  integration  between  acquisition 
and  logistics  for  total  system  life-cycle,  2)  incentives,  and  3)  performance  goals. 


Generally,  the  contracting  agency  seeks  to  improve  perfomaance  throughout  the  life 
of  a  weapon  system  in  some  measurable  without  dictating  the  specific  methods 
of  performance.  Moreover,  the  agency  is  willing  to  praide  incentives  to  the  contractor 
to  meet  these  performance  objectives.  PBL  integration  replaces  the  practice  of  attempt¬ 
ing  to  define  specific  methods  of  operation  by  describing  desired  results  and  uses 
incentives  to  ensure  success. 

The  official  definition  from  the  Defense  Acquisition  Guidebook  (2004)  is: 

Pferformance-Based  Lqgjstics  is  the  purchase  of  support  as  an  int^jated, 
affordable,  performance  package  designed  to  optimize  system  readiness 
and  meet  performance  goals  for  a  weapon  system  through  long-term 
support  arrangements  with  clear  lines  of  authority  and  responsibility. 
Application  of  Perfoimance-Based  Logistics  may  be  at  the  system,  sub¬ 
system,  or  major  assembly  level  depending  on  program  unique  circum¬ 
stances  and  appropriate  business  case  analysis,  (p.  5.3) 

The  Navy  provides  the  inclusive  term  “provider”  ^^hich  demonstrates  that  functions 
can  be  performed  by  any  entity  A  addition  to  the  Navy  definition  is  the  inclusion 
of  the  terni  “empowered,”  implying  that  additional  power  in  decision  making  is  granted 
to  the  provider.  This  illustrates  the  move  aw^  from  centrally  controlled  performance  to 
more  localized  performance.  Below  is  the  Department  of  the  Navy  (2003)  definition. 

A  PBL  strategy  is  an  agreement,  usually  long  term,  in  which  the  pro¬ 
vider  (organic,  commercial,  and/or  public/private  partnership)  is 
incentivized  and  empowered  to  meet  overarching  customer  oriented 
perfonnance  requirements  (reliability,  availability,  etc.)  in  order  to  im¬ 
prove  product  support  effectiveness  while  reducing  TOC.  (p.  1) 

The  Army  elevates  PBL  to  a  strategy.  While  not  focused  on  the  customer,  per  se, 
the  definition  does  link  PBL  efforts  to  the  purchase  of  readiness. 

A  strata  for  we^n  system  product  siqiport  that  employs  the  purchase 
of  support  as  an  integrated  performance  package  designed  to  optimize 
system  readiness.  It  meets  performance  goals  for  a  we^n  system  through 
a  support  stmcture  based  on  performance  agreements  with  clear  lines  of 
authority  and  responsibility.  (Hill  &  Hamerlinck,  2003,  p.  7) 

The  Air  Force  does  not  use  the  term  PBL.  Instead  the  Air  Force  uses  Total  System 
Support  Responsibility  (TSSR),  Total  System  Performance  Responsibility  (TSPR),  Flex¬ 
ible  Sustainment,  and  Total  Life-Cycle  System  Support.  While  the  terms  are  different, 
the  concepts  are  the  same.  Air  Force  programs  focus  on  systemwide  support  to  provide 
total  system  sustainment  and  system  level  readiness.  The  implications  of  the  Air  Force 
definitions  are  that  systems  should  be  acquired  and  sustained  for  the  long-term.  The 
Air  Force  concept  is  similar  to  definitions  used  by  the  Deputy  Under  Secretary  of 
Defense  for  Logistics  and  Materiel  Readiness  and  Naval  Air  Systems  Command 
(NAVAIR). 
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Since  Industry  does  not  specifically  use  PBL  terminology,  no  definition  of  PBL 
exists.  Industry  uses  the  term  Supply  Chain  Management  (SCM)  to  describe  efforts 
similar  to  PBL.  In  general,  the  logistics  support  function  involves  inclusive  contracts 
with  service  providers  to  provide  a  level  of  service  that  is  required  by  the  acquiring 
company.  Traditional  commercial  products  seldom  require  the  degree  of  sophisti¬ 
cated  systems  as  the  DoD.  However,  commercial  high  technology  products  do  in¬ 
volve  high  levels  of  sophistication  and  exact  specifications.  In  some  cases,  items 
that  require  on-going  logistical  support  and  repair  are  outsourced  with  a  third  party 
managing  the  entire  process. 

We  developed  the  following  comprehensive  definition  of  PBL  to  capture  the  vari¬ 
ous  tenets  discussed  above. 

An  int^juted  acquisition  and  sustainment  strategy  for  enhancing  weapon 
system  capability  and  readiness  where  the  contractual  mechanisms  will 
include  long-term  relationships  and  appropriately  structured  incentives 
with  service  providers,  both  organic  and  non-oiganic  to  support  the  end 
user’s  (warfighter’s)  objectives. 


DRIVERS  OF  PBL 

In  general,  DoD  focuses  on  developing  programs  designed  to  enhance  perfor¬ 
mance  and  reduce  total  system  cost  over  the  life  of  a  weapon  system.  The  desire  by 
DoD  to  change  the  way  they  conduct  business  led  to  the  PBL  initiative.  Our  inter¬ 
views  revealed  numerous  reasons  for  the  adoption  of  PBL.  We  report  seven  primary 
drivers  for  PBL  in  Table  2. 

Inherent  in  these  drivers  for  PBL  is  both  the  perception  and  the  reality  that  weapon 
systems  are  expensive  to  maintain,  difficult  to  upgrade  with  new  technology,  and 
take  a  long  time  to  diflFuse  to  the  field.  Moreover,  this  is  also  true  for  the  repair  and 
maintenance  of  fielded  (new  and  legacy)  systems.  These  PBL  drivers  focus  on  chang¬ 
ing  the  current  environment  by  suggesting  strategic  directions  for  the  future. 


tABLi  1.  DRIVERS  FOR  PBL 


DRIVERS  FOR  PERFORMANCE-BASED  LOGISTICS 


1.  Rising  cost  of  maintenance,  operations,  and  support  for  new  and  legacy  missile  systems. 

2.  Needed  tool  for  Logistics  Transformation  and  other  actions  required  by  Congress. 

3.  Needed  reduction  of  customer  wait  time  in  support  of  the  war  fighter. 

4.  Needed  modernization  of  weapon  systems  to  enhance  combat  capability. 

5.  Needed  solutions  to  weapon  obsolescence  problems. 

6.  Documented  savings  from  commercial  logistics  support  operations. 

7.  Documented  improvements  from  implementation  of  performance— based  acquisition. 
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INFRASTRUCTURE  CHANGES 

A  major  challenge  for  conversion  to  a  PBL  environment  is  to  adopt  business 
practices  more  common  in  commercial  organizations.  To  meet  the  objectives  of 
PBL,  both  government  and  Industry  must  agree  on  business  practices  that  provide 
the  greatest  value  for  all  parties. 

Our  research  reveals  that  a  move  to  PBL  requires  several  infrastructure  changes. 
To  keep  the  study  at  a  strategic  level,  we  focused  on  the  need  to  change  the 
culture  of  the  implementing  organization  since  it  was  the  recurring  theme  through¬ 
out  our  PBL  research.  Organizational  culture  is  “a  pattern  of  beliefs  and  expec¬ 
tations  shared  by  organizational  members”  (Hellriegel,  Slociun,  &  Woodman,  1986). 
These  shared  beliefs  and  expectations  determine  the  behavior  of  the  members  of 
the  organization.  Changing  organizational  culture  is  complicated  by  the  fact  that 
people  tend  to  surround  themselves  with  others  of  like  opinions  and  values,  thus 
reinforcing  their  common  beliefs  and  expectations  (Schein,  1981). 

For  example,  an  AMC  HQ’s  team  identified  21  issues  that  must  be  addressed 
prior  to  PBL  implementation.  One-third  of  these  issues  reflect  a  culture  or  belief 
that  would  not  be  supportive  of  PBL  implementation.  Table  3  presents  several 
examples  obtained  from  our  interviews  of  Old  Culture  beliefs  that  are  juxtaposed 
with  PBL  examples  of  New  Culture  or  new  w^s  of  doing  business.  Several  models 
for  successful  change  management  can  be  found  in  the  management  literature 
(Camm,  Drezner,  Lachman,  &  Resetar,  2001).  There  are  also  excellent  examples 
of  government  success  in  changing  the  culture  of  specific  organizations. 


To  meet  the  obie<tives  of  PBL,  both  government 
and  Industry  must  agree  on  business  pratthes 
that  provide  the  greatest  value  for  all  parties. 


For  instance,  in  September  2002,  GAO  (2002)  issued  a  report  that  stated,  “DLA 
does  not  provide  a  ‘single  face’  to  its  customers  for  addressing  their  issues.”  Customers 
are  “sometimes  confused  over  whom  to  call  and  reported  difficulties  with  getting  in 
touch  with  the  right  person  to  resolve  their  problems”  (p.  21).  GAO  recommended 
DLA  create  a  single  face  to  customers  to  improve  customer  satisfaction.  DLA  has 
since  implemented  a  customer  relationship  management  (CRM)  program  to  learn 
more  about  its  customers’  needs  and  behaviors.  They  have  also  realigned  the  DLA 
organization  structure.  They  now  have  functional  field  chiefs  reporting  to  directors 
at  headquarters  and  established  a  new  Customer  Operations  Directorate. 
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TABLE  3.  CULTURE  EXAMPLES 

COMPARISON  OF  CULTURE  EXAMPLES 

The  C-1 7  aircraft  is  the  focus  of  a  Boeing  -  Air  Force 
partnership.  They  do  joint  off  sites  and  work  specifically 
on  their  relationship.  They  have  joint  weekly,  monthly, 
block,  etc.,  meetings  and  reviews.  Every  employee  who 
works  on  the  C-1 7  wears  a  plastic  card  the  size  of  their 
badge,  imprinted  with  the  partnership  agreement  signed 
by  Boeing  and  Air  Force  leaders. 

-f  Arms  length,  adversarial  relationship  between 
government  and  contractor. 

>  All  communications  in  writing  to  create  an  audit  trail. 

4-  Interact  as  little  as  possible,  conduct  bi-annual 
performance  reviews. 

4-  Maintain  objectivity  don't  get  too  close  to  the 
contractor. 

4>  Contractor  driven  by  profit  motive  vs.  nation’s 
defense. 

♦  Government  close  holds  information. 

NAVSEA  established  an  e-marketplace  using  a 
one-page  flowchart  showing  what  it  wanted  its 
electronic  services  procurement  system  to  look  like. 

The  five  steps  represented  the  full  operating  capability 
{FOC)  of  the  desired  system,  with  the  extensions  and 
clouds  being  areas  for  future  scalability  in  the  eventual 
system.  The  Navy  simply  handed  the  flowchart  to 
potential  vendors  and  asked  them,  “How  much  of  this 
picture  can  you  deliver  and  at  what  price?” 

(IBM  -  Seaport  Study,  p.  18) 

4“  Lengthy  statements  of  work  developed  by 
government  requiring  office— with  an  attempt  to 
document  every  possible  situation,  process, 
regulation,  milspec,  service,  and  government 
expectation  for  the  bidders. 

4“  Independent  government  estimates. 

>  Elaborate  processing  of  Statement  of  Work  through 
technical  data,  system  engineering,  legal,  etc., 
all  with  organization-specific  word  requirements. 

Air  Force  Joint  Surveillance  Target  Attack  Radar  System 
(JSTARS)  Total  System  Support  Responsibility  (TSSR) 
partnership  has  multiple  agreements  in  place 
supporting  the  sustainment  of  JSTARS.  In  most  cases, 
these  agreements  stand  alone— they  are  not  part  of  the 
contract  between  Northrop  Grumman  Corporation 
(NGC)  and  the  Air  Force.  The  Partnering  Agreement 
(PA)  between  NGC  and  the  Warner  Robbins  Air 

Logistics  Center  (WR-ALC)  has  been  incorporated  into 
the  prime  TSSR  contract  as  the  guiding  basis  for  the 

Air  Force  providing  the  depot-performed  workloads  to 
the  contractor. 

4-  Finger  pointing  between  government  and  suppliers 
over  delays  and  cost  increases, 

♦  Request  for  Proposal  describes  services  and 
scope  of  work  in  great  detail. 

4-  Numerous  change  orders  as  soon  as  work  starts 
and  RFP  omissions  are  identified. 

4-  Government  defines  service  delivery  means  and 
process  through  inclusion  of  government 
regulations  and  directives. 

4^  Contract  administration  role  vs.  partner  role. 

>  Only  acceptable  relationship  is  a  contractual  one. 

Sikorsky  Aircraft  Corporation  (SAC)  is  working 
side-by-side  with  Corpus  Christi  Army  Depot  (CCAD)  to 
reduce  repair/overhaul  turnaround  time  for  the  H-60. 

This  joint  collaboration  has  improved  business 
processes,  depot  repair  methodology,  and  more 
responsive  product  support,  with  only  four  contractor 
jobs  directly  attributable  to  the  partnership. 

♦  Expert  role  assigned  to  government  employee. 

4^  Use  of  design  specifications  where  the  government 
tells  the  contractor  howto  provide  the  service. 

♦  Contractors  in  the  government  workplace  viewed 
as  personal  service. 

4“  Quality  assurance  processes  defined  by 
government  specialists. 

4>  Government  employee  relies  on  guidance  from  HQ. 

The  Navy  Inventory  Control  Point  (NAVICP)  has  an 
F/A-18E/F  Integrated  Readiness  Support  Teaming 
(FIRST)  prime  contract  with  Boeing  under  which 

Navel  Air  Depot  (NADEP)  North  Island  performs  depot 
repair  services  to  Boeing  as  a  subcontractor.  Boeing 
provides  funding,  repairable  units,  repair  parts, 
obsolescence  management,  and  shipping.  The  NADEP 
North  Island  provides  touch  labor,  facilities,  technical 
data,  equipment,  production  engineering,  and 
packaging.  Fifty-seven  government  jobs  were  created 
or  sustained  by  this  partnership. 

4“  Contractors  are  taking  jobs  away  from  government 
workers. 

4"  Government  is  buyer  of  services,  not  seller. 

4-  All  payments  to  government  are  deposited  in  the 
U.S.  Treasury  account. 

4*  Private  sector  cannot  use  government  facilities  and 
equipment  to  perform  work. 
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GUIDELINES  FOR  IMPLEMENTING  PERFORAAANCE-BASED  LOGISTICS 

Based  on  our  research  and  the  incorporation  of  the  findings  from  RAND  and  the 
Aberdeen  Group  (Camm,  Drezner,  Lachman,  &  Resetar,  2001;  Leahy,  2003),  we  propose 
the  following  six  guidelines  to  successfully  implement  PBL: 

1 .  Assign  responsibilities  dearly  throughout  the  firm — ^Blanket  statements  about 
policy  changes  that  imply  that  PBL  is  everyone's  responsibility  are  typically 
ineffective.  Experience  suggests  that  anything  that  is  everyone’s  responsibility 
is  no  one’s  responsibility.  To  vaiying  degrees,  the  Navy,  Air  Force,  and  DLA 
all  address  this  issue.  Each  of  these  organizations  requires  that  responsibility 
for  the  success  of  ary  PBL  program  be  assigned  to  a  specific  unit. 

2.  Design  metrics  to  motivate  the  right  behavior — The  cliche  successful  firms 
manage  what  can  be  measured  can  be  overstated.  Nevertheless,  RAND  found  that 
proactive  firnis  do  rely  on  metrics  as  the  foundation  for  managing  improvement 
(Camm,  Drezner,  Lachman,  &  Resetar,  2001).  Accordingly,  metrics  designed  to 
motivate  the  right  behavior  must  be  carefully  crafted  and  applicable  across  the 
entire  organization.  Effective  metrics  must  induce  the  decision  maker  to  pursue 
[organizational]  goals,  be  compatible  with  the  constraints  that  the  decision  maker 
faces  in  each  setting,  be  easy  to  collect  and  verify,  and  be  mutual^  understood 
and  accepted  by  the  decision  malsr  and  oversight  authority  (Hellriegel  et  al.,  1986). 
For  instance.  Naval  Inventory  Control  Point  (NAVICP)  is  responsible  for  the  Navy 
PBL  program  and  NAVICP  Operations  Research  (OR)  Group  is  focused  on  de¬ 
veloping  appropriate  performance  metrics  for  logistical  operations. 

Defining  the  right  PBL  metrics  is  difficult  for  both  government  and  contractors. 
NAVICP  is  using  its  OR  group  to  answer  these  questions.  Initial^,  it  may  be  easy 
for  contractors  to  exceed  expectations  and  improve  performance.  After  the  initial 
changes  take  place,  it  becomes  increasingly  difficult  to  continue  to  gain  higher 
levels  of  performance.  Contractors  and  government  employees  predicted  future 
difficulties  in  this  area.  For  instance,  one  Navy  contractor  indicated  that  he  is 
currently  engaged  in  negotiations  for  more  difficult  metrics  while  his  firm’s  current 
performance  is  within  acceptable  performance  expectations.  At  the  same  time, 
NAVICP  is  attempting  to  quantify  their  requirements.  For  example,  NAVICP  might 
purchase  a  three-day  delivery  when  a  ten-day  delivery  would  be  acceptable. 
Finally,  metrics  and  incentives  should  be  designed  simultaneously  to  ensure  that 
performance  is  measured  correct^  and  rewarded  appropriately. 

3 .  Manage  failures  to  limit  disincentives  for  risk-taking— Eailure  is  part  of  the  learning 
process.  The  term  failing  forward,  describes  the  process  of  “creating  forward 
momentum  with  the  learning  derived  from  failures”  (Leonard-Barton,  1995).  While 
most  commercial  firms  understand  feihng  forward,  we  found  little  insight  into  hew  to 
implement  the  concept  in  DoD  The  PBL  requires  interdisciplinary  organizations  and 
teams,  consisting  of  professionals  with  advanced  interpersonal,  analytic,  and  computer 
skills,  and  requires  knowledge  of  contracting,  logistics,  funds  managanent,  metrics. 
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and  organizational  effectiveness  and  efficiency.  It  also  requires  building  relation¬ 
ships  with  contractors  and  operating  from  a  holistic  view  of  the  organization. 

4.  Develop  a  supportive  organizational  context  for  tools — ^These  tools  include 
middleware  to  standardize  decision  making  based  on  legacy  system  output  and 
tracking  systems  to  document  performance  improvements  and  lessons  learned  across 
the  organization.  The  Warner  Robins-Air  Logistics  Center(WR-ALC)  uses  Supply 
Chain  Common  Operating  Picture  (SCCOP)  to  provide  a  common  operational 
view  of  the  total  supply  chain  and  specific  details  on  all  factors  that  affect  weapon 
system  availability.  Each  data  element  is  obtained  from  the  designated  authorita¬ 
tive  source  for  the  information.  This  is  accomplished  through  the  retrieval,  displ^, 
and  integration  of  information  captured  from  multiple  legacy  data  sources. 

5.  Manage  relationships  with  stakeholders — Continuing  communication  v^th  stake¬ 
holders  is  one  way  to  gain  their  support.  In  the  case  of  environmental  management, 
Procter  &  Gamble  invested  time  to  train  state  regulator  personnel  on  issues  relevant 
to  the  Industry  The  DoD  Inspector  General  (IG)  is  a  similar  regulator  that  may 
have  some  difficulty  accepting  PEL  required  changes  in  contract  management  and 
administration.  The  DLA  Customer  Relationship  Management  (CRM)  Office  pro¬ 
vides  a  consolidated  approach  to  developing  and  delivering  information  related  to 
DLA  business  goals  to  key  stakeholders  and  DLA  customers.  Using  an  Integrated 
Product  Team  (IPT)  network  of  customer-touch  points,  strategic  level  information 
at  headquarters  (from  public  affairs  offices  and  current  DLA  publications  staff)  is 
integrated  with  field  level  activities.  The  CRM  office  then  develops  content  and 
tools  to  provide  the  needed  message  to  customers. 

6.  Benchmark  to  promote  continuous  improvement — In  order  to  find  how  well 
initiatives  are  working,  compare  results  through  the  benchmarking  process.  This 
requires  finding  the  best  practices  in  Industry  and  government  and  ^e  identifica¬ 
tion  of  those  firms  and  agencies  that  are  the  best  perfoimer  for  a  specific  activity. 
Utilizing  these  findings,  identify  the  gaps  and  develop  a  plan  to  close  the  gaps. 
In  order  to  be  successful  in  implementing  PEL,  such  tenchmarking  and  improve¬ 
ment  processes  need  to  become  a  habit  for  an  organization  rather  than  ad-hoc 
actions.  Many  of  the  organizations  that  we  interviewed  prodded  insight  into  how  the 
PEL  transition  was  linked  to  developing  Lean  processes.  At  both  the  Air  Force  and 
Navy,  the  benchmarks  were  developed  based  on  the  leaner,  more  efficient  organi¬ 
zations  and  then  used  to  become  the  basis  for  continued  improvement.  The  Air  Tbrce 
maintains  an  on-going  contract  with  RAND  to  provide  benchmarking  services 

These  six  guidelines  are  derived  from  a  variety  of  lessons  learned.  We  have 
synthesized  our  research  and  the  findings  of  other  researchers  to  provide  a  starting 
point  for  the  implementation  of  PEL.  Each  organization  must  create  an  atmosphere  of 
trust  and  commitment  with  both  its  customers  and  suppliers.  The  organization  must 
focus  on  its  core  competencies  and  create  relationships  or  do  strategic  sourcing  with 
organizations  to  enhance  the  value  of  offerings  for  customers. 
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CONCLUSIONS 

We  have  reviewed  the  best  practices  and  lessons  learned  in  DoD  and  Industry  for 
managing  and  sustaining  a  complex  system  in  a  performance-based  environment. 
Specificalfy,  we  identified  issues  and  key  criteria  for  developing  strat^es  and  policies 
to  define  the  required  relationships,  contractual  mechanisms,  and  incentives  to  achieve 
the  objectives  of  PBL  in  government.  Our  research  efforts  resulted  in  the  development 
of  a  definition  of  PBL  to  be  used  to  guide  deployment  efforts.  We  also  identified  the 
drivers  for  PBL  and  the  infrastructure  changes  needed  to  successfully  use  PBL  in 
government.  Based  on  our  research  findings  and  those  of  previous  related  studies  in 
government  and  Industry,  we  proposed  six  guidelines  to  successfully  implement  PBL. 


...future  researth  diretted  toward  the 
development  of  methodologies  and  algorithms  for 
creating  performame  spedfUations  wiii  enhame 
the  implementation  and  aiteptante  of  PBl 
approathes  in  government. 


This  research  also  suggests  some  directions  for  future  efforts  needed  to  successfully 
implement  PBL  in  government.  First,  an  educational  program  to  clarify  the  understand¬ 
ing  and  comprehension  of  the  definition,  scope,  and  purpose  of  PBL  will  mitigate  some 
myths  and  fears  about  its  use.  Second,  each  of  the  six  guidelines  requires  further  research 
to  develop  specific  policies,  procedures,  and  measures  for  their  use  and  effectiveness. 
Third,  while  the  conceptual  ftamework  for  PBL  envisions  an  oiganizationwide  adoption, 
in  actual  practice  it  is  more  incremental  in  nature.  Therefore,  further  research  is  needed 
to  develop  a  quantitative  method  to  rank  order  projects  that  are  candidates  for  an  early 
adoption  of  PBL. 

Fourth,  since  the  PBL  approach  emphasizes  performance,  appropriate  performance 
expectations  need  to  be  specified.  This  requires  the  development  of  the  performance 
specifications  needed  fi-om  the  customers  using  the  systems  being  acquired  and  sus¬ 
tained.  Therefore,  future  research  directed  toward  the  development  of  methodologies  and 
algorithms  for  creating  performance  specifications  will  enhance  the  implementation  and 
acceptance  of  PBL  approaches  in  government.  Finally,  development  and  implementation 
of  performance-based  incentives  that  include  some  form  of  innovation  and  technology 
enhancement  will  be  required  to  realize  the  full  benefits  of  PBL.  While  this  research  has 
resulted  in  creating  suitable  guidelines  for  implementing  PBL,  more  research  is  needed  to 
make  it  a  reality 
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A  PERFORMANCE-BASED 
TECHNOLOGY 
ASSESSMENT 
METHODOLOGY 
TO  SUPPORT  DOD 
ACQUISITION 

DR.  SHERRY  MAHAFZA  DR.  PAUL  COMPONATIOH 
AND  DR.  DONALD  TIPPEH 


Many  weapon  system  failures  are  attributed  to  premature  transfer  of 
technology  to  operational  systems.  Insufficient  measures  of  assessing 
technology  readiness  are  major  contributors  to  such  failures.  This  paper 
presents  a  methodology  to  measure  the  performance  risk  of  technology 
in  order  to  determine  its  transition  readiness.  This  methodology  is  referred 
to  as  Technology  Performance  Risk  Index  (TPRI).  The  TPRI  can  track 
technology  readiness  through  a  life  cycle,  or  it  can  be  used  at  a  specific 
time  to  support  a  particular  system  milestone  decision.  The  TPRI  is  computed 
using  the  performance  requirements,  the  Degree  of  Difficulty  (DD),  and 
the  unmet  performance.  These  components  are  combined  in  a  closed- 
loop  feedback  manner  to  analytically  calculate  the  performance  risk.  TPRI 
is  illustrated  by  an  example  using  published  system  requirements  data. 


Since  World  War  II,  the  United  States  Armed  Forces  have  maintained  a 
technological  advantage  over  adversaries.  Today,  the  military  is  facing  continued 
threats  that  require  an  accelerated  pace  of  technology  development  amid  global 
proliferation  of  military  technologies  (Lukens,  2003).  The  Department  of  Defense 
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(DoD)  has  estimated  a  need  for  $50  billion  dollars  for  missile  defense  research 
and  development  over  years  2004-2009  (General  Accounting  Office  [GAO], 
2003);  however,  this  requirement  must  be  balanced  with  other  funded  programs. 

The  demands  to  support  additional  operational  tempos,  higher  maintenance  costs 
for  aging  weapon  systems,  and  higher  system  acquisition  costs,  limit  the  available 
funding  for  new  technology  development.  These  increasing  demands  compete  for  the 
same  money  used  for  research  and  development  of  technology,  and  often  the  technology 
budget  is  further  reduced.  Additionally,  the  impact  of  company  buyouts  has  reduced 
the  military  industrial  research  base  in  the  United  States  from  21  companies  in  1993 
to  5  companies  in  2002  (Linster,  Slate,  &  Waller,  2002).  Consequently,  this  has  resulted 
in  substantial  reduction  in  Independent  Research  and  Development  (IR&D)  activities. 
In  tody’s  DoD  environment,  it  is  important  that  investments  in  technology  are 
successfully  transitioned  to  operational  military  systems. 


Changing  to  a  iapabiUty-based  atguisition 
strategy  is  another  intiitator  of  the 
signifUante  of  tethnoiogy. 


Changing  to  a  capability-based  acquisition  strategy  is  another  indicator  of  the 
significance  of  technology.  Mr.  Pete  Aldridge  identified  in  a  memo  (Aldridge,  2002) 
to  the  Secretary  of  Defense  his  intent  to  accelerate  the  flow  of  technology  to  the 
warfighter.  Later,  Mr.  Aldridge  (2003)  announced  his  goal  to  initiate  high-leverage 
technologies  to  create  the  warfighting  capabilities  and  strategies  of  the  future.  There¬ 
fore,  it  is  imperative  that  the  technology  meets  maturity  and  performance  requirements, 
prior  to  being  transitioned  to  the  acquisition  system.  Furthermore,  a  capability-based 
acquisition  strategy  will  allow  acquisition  programs  to  pull  advanced  technology  into 
systems  faster,  thus  fielding  systems  with  advanced  technologies  to  the  warfighter 
faster.  The  capability-based  acquisition  cycle  also  means  that  requirements  will  evolve 
faster,  which  mandates  close  monitoring  of  systems’  requirements  and  the  technologies 
we  use  to  meet  these  requirements. 


THE  PROBLEM 

Development  of  new  defense  technologies  within  the  DoD  is  a  multi-dimensional 
problem.  First,  DoD  must  resolve  issues  that  result  from  immature  technologies 
transition.  The  General  Accounting  Office  (GAO)  has  stated  that  immature  technology 
transition  is  the  leading  cause  of  weapon  system  problems  (GAO,  1999).  An  important 
factor  in  the  success  of  a  new  weapon  system  is  ensuring  that  technologies  are  mature 


270 


prior  to  being  integrated  (GAO,  1999).  Second,  the  creation  of  parallel  paths  for 
the  development  of  technology  and  the  development  of  an  acquisition  weapon 
system  has  diluted  the  link  between  technology  and  system  performance  require¬ 
ments.  The  technologist  has  responsibility  for  managing  the  development  of  the 
technology,  while  the  weapon  system  acquisitionist  has  responsibility  for  the 
development  of  the  weapon  system.  Unfortunately,  the  technologist  has  different 
goals,  environments,  and  perspectives  than  the  system  acquisitionist  (McGrath, 
2003).  The  original  reasoning  behind  this  deliberate  separation  is  that  it  allows 
the  acquisitionist  to  focus  on  meeting  requirements  for  the  system  development, 
while  providing  the  technologist  an  environment  to  explore  capabilities  of  the 
technology.  An  unforeseen  result  of  this  separation  is  that  two  conflicting  drives 
of  motivation  are  generated. 

The  technologist  is  motivated  to  transition  technologies  into  weapon  systems.  Thus, 
technologists  are  optimistic  on  the  maturity  assessment  of  their  technology.  The 
acquisitionist  is  motivated  to  meet  system  requirements,  and  often  uses  a  risk  adverse 
approach  for  the  design  process.  Consequently,  the  acquisitionist  is  more  likefy  to 
underestimate  the  maturity  of  new  technologies.  This  forces  the  technologist  to  focus 
on  risk  mitigation.  These  conflicting  motivations  justify  the  need  for  an  objective 
methodology  to  assess  a  technologies  fit  with  system  performance  requirements.  As 
a  technology’s  maturity  increases,  the  criticality  for  decision  support  tools  to  deter¬ 
mine  transition  readiness  is  also  increased.  Hence,  a  common  understanding,  between 
the  technologist  and  the  acquisitionist,  is  needed. 


PROPOSED  METHODOLOGY 

One  approach  to  develop  a  common  understanding  of  technology  readiness  is  to 
utilize  a  modified  version  of  Garvey’s  system  performance  risk  index  (Garvey  & 
Cho,  2003).  The  threshold  value  of  a  Teclmical  Performance  Measure  (TPM)  divides 
performance  into  acceptable  and  unacceptable  risk  regions.  In  this  manner,  it  is  the 
goal  of  a  system  developer  to  reach  the  acceptable  performance  risk  region.  To  get 
into  the  acceptable  performance  risk  region,  the  technology  must  meet  or  exceed  the 
identified  TPM  threshold.  Garvey  provided  guidelines  for  calculating  the  achieved 
performance. 

The  proposed  methodology  to  assess  technology  readiness  proposed  in  this  paper 
is  referred  to  as  the  Technology  Performance  Risk  Index  (TPRI).  The  index  is  based 
on  the  system’s  performance  requirements  and  the  ability  of  the  technology  to  achieve 
that  performance.  The  achieved  performance  is  normalized  so  multiple  requirements 
can  be  assessed  simultaneously.  A  condensed  solution  for  the  two  cases  of  the  re¬ 
quired  performance  behaviors  is  then  calculated.  In  the  first  case,  perfonnance  must 
decrease  to  meet  the  established  TPM  threshold.  The  achieved  performance  is  com¬ 
puted  as  the  inverse  of  the  percentage  of  the  TPM  threshold  that  the  measured  per¬ 
formance  represents.  The  achieved  performance,  A.^  at  time  i,  for  TPM  j  is  calculated 
by: 
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Ay  =  min 


threshold 


my 


,1 


Equation  1 


where  my  is  the  measured  performance.  Examples  of  this  type  of  required  per¬ 
formance  behavior  are  weight  constraints  or  mean  time  to  repair  TPMs.  In  the 
second  case,  performance  must  increase  to  meet  the  threshold  and  A^.,  at  time 
/,  for  the  jth  TPM  is  computed  using;  ^ 


A 


v 


min 


— ~ — ,l| 
threshold  J 


Equation  2 


Again,  my  is  the  measured  performance.  In  the  increasing  performance  behavior, 
the  achieved  performance  is  equivalent  to  the  percentage  that  the  measured  performance 
represents  of  the  TPM  threshold.  Two  examples  of  increase  perfonnance  behavior 
TPMs  would  include  number  of  units  available  or  mean-time-between-failures. 

Garvey  defines  the  system  performance  risk  index  as  the  amount  of  the  remaining 
unmet  performance  relative  to  meeting  a  set  of  identified  TPMs.  Emphasis  is  placed 
on  the  unmet  performance  as  issues  for  management  to  focus  and  resolve  with  pri¬ 
orities  and  allocation  of  resources.  Although  Garvey’s  method  provides  a  quantitative 
measure  of  meeting  a  set  of  established  requirements,  it  inherently  lacks  the  inclusion 
of  a  tme  risk  measure.  More  specifically,  in  Garvey’s  approach,  unmet  performance 
does  not  indicate  what  it  will  take  to  reach  the  acceptable  risk  region.  It  merely  provides 
a  measure  of  progress  achieved  at  a  certain  time. 


Garvey  defines  the  system  performame  risk 
index  as  the  amount  of  the  remaining 
unmet  performame  relative  to  meeting 
a  set  of  identified  TPil/ls. 


This  article  presents  a  research  effort  to  address  the  linkage  between  technolo¬ 
gies,  system  performance  requirements  and  risks  in  the  DoD  environment.  This 
methodology  is  used  to  measure  risk  associated  with  satisfying  an  identified  set 
of  TPMs  for  a  given  system.  The  TPRI  provides  information  regarding  perfor¬ 
mance  risk  associated  with  a  technology  to  support  the  decisions  of  whether  or  not 
a  certain  technology  is  ready  to  be  transitioned  to  an  acquisition  system.  The  TPRI 
offers  a  measure  of  the  actual  performance  risk  of  a  technology  by  providing  an 
assessment  of  a  given  technologies  ability  to  achieve  performance  requirements. 
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Thus,  a  common  understanding  of  performance  risks  between  technologist  and 
acquisitionist  is  accomplished.  The  impact  of  the  TPRI  can  be  described  using 
the  following  analogy:  a  thermostat  measures  actual  outside  temperature,  yet  to 
a  human,  the  wind  chill  factor  is  of  more  concern.  In  this  same  manner,  the  TPRI 
is  like  the  wind  chill  factor,  as  it  affords  valuable  information  that  allows  insight 
into  the  technology  that  would  not  be  obtained  from  individual  metrics. 

Performance-based  requirements  criteria  are  typically  established  by  an 
acquisitionist.  Typically,  these  criteria  and  the  basis  for  measurement  are  identified 
in  an  agreement  with  the  technologist.  This  serves  as  a  basis  consisting  of  the  indi¬ 
vidual  requirements  and  associated  threshold  values  a^inst  which  technologies  will 
be  evaluated.  This  information  is  utilized  to  compare  against  the  measured  level  of 
performance  achieved. 

The  TPRI  extends  Garvey’s  methodology  to  include  a  measure  of  performance  risk 
so  that  acquisitionists  can  have  access  to  information  that  will  assist  in  setting  priorities 
and  allocating  resources  (Garvey  &  Cho,  2003).  For  this  purpose,  the  Degree  of 
Difficulty  (DD)  is  the  metric  utilized  TPRI.  The  DD  metric  (Mankins,  1998  & 
2002)  provides  a  measure  of  anticipated  risk,  ranging  from  low  level  risk  to  high 
level  risk,  and  can  be  considered  as  a  probability  of  failure  in  regards  to  the  technology 
achieving  the  objectives. 

In  TPRI,  the  DD  metric  is  modified  to  assign  a  numerical  value  to  each  of 
Mankins’  defined  levels.  This  provides  a  quantitative  means  to  combine  risks 
across  various  requirements.  In  this  context,  the  DD  metric  is  bounded  by  zero 
and  one.  Table  1  provides  a  summary  of  each  of  the  DD  levels  with  anticipated 
risk,  and  the  numerical  value,  as  well  as  the  theoretical  boundary  levels.  The 
lower  boundary  with  a  DD  of  zero  indicates  there  is  no  risk  involved  in  meeting 
the  objectives  and  is  guaranteed  success.  The  DD  at  level  1  has  a  very  low  risk 
and  is  assigned  a  0.1  value;  a  DD  at  level  2  has  an  anticipated  moderate  risk 
and  has  an  assigned  value  of  0.3,  while  the  DD  at  level  3  has  a  high  risk,  with 


TABLE  !•  DEGREE  OF  DIFFICULTY  (DD)  LEVELS  WITH  RISKS 
AND  ASSIGNED  NUMERICAL  VALUES 


Degree  of  Difficulty 

Risk  Level 

DD  Value 

Level  0 

(Theoretical  Lower  Bound) 

No  Risk;  Guaranteed  Success 

0.0 

Level  1 

Very  Low  Risk 

0.1 

Level  2 

Moderate 

0.3 

Level  3 

High 

0.5 

Level  4 

Very  High 

0.7 

Level  5 

Extremely  High  Risk,  Requiring  a  Fundamental  Breakthrough 

0.9 

Level  6 

(Theoretical  Upper  Bound) 


Guaranteed  Failure 


1.0 


a  0.5  value.  A  technology  with  very  high  risk  is  identified  as  DD  level  4  and 
has  a  0.7  value.  In  the  case  that  a  technology  has  a  high  expectation  of  failure, 
requiring  a  fundamental  breakthrough,  is  assigned  a  DD  level  5  with  a  0.9 
numerical  value.  In  addition,  the  theoretical  upper  bound  of  the  DD  metric 
indicates  the  highest  level  of  risk  with  no  anticipated  success  (guaranteed  fail¬ 
ure),  and  is  assigned  a  value  of  1. 

A  major  attribute  of  TPRI  is  that  it  accounts  for  unmet  performance  and  associated 
risk  to  quantify  effects  upon  the  achieved  performance.  Risk  is  defined  as  a  measure 
of  the  probability  and  the  severity  of  adverse  effects  (Haimes,  1998).  The  TPRI  is 
applicable  to  the  technical  risk  form;  referred  to  as  performance  risk.  The  TPMs 
provide  a  gauge  of  technical  progress  measured  against  satisfying  an  identified 
set  of  thresholds  pertaining  to  performance  requirements.  The  TPRI  correlates 
the  performance  risk  associated  with  a  technology  not  meeting  the  threshold  value 
of  a  TPM. 


TPRI  METHODOLOGY 


To  formulate  the  TPRI  model,  the  unmet  performance  and  DD  are  combined  to 
adjust  the  achieved  performance.  The  TPRI  is  expressed  mathematically  as: 


TPRI  =  \ - - 

\-¥{\-A)*DD 


Equation  3 


The  unmet  performance,  1—A,  is  a  measure  of  the  severity  component  of  risk,  while 
DD  signifies  the  probability  component  of  risk.  These  two  components  are  combined 
to  formulate  a  measure  of  performance  risk  of  the  unmet  performance-based  on  a 
closed  loop  feedback  mechanism.  This  index  provides  a  measure  of  performance  risks 
to  meet  the  TPM  threshold  in  an  effort  to  reach  the  acceptable  performance  risk 
region  goal. 


4  maior  attribute  of  TPRI  is  that  it  at€ounts  for 
unmet  performame  and  assodated  risk  to 
quantify  effetts  upon  the  athieved  performame. 


The  behavior  of  the  TRPI  model  is  depicted  in  Figure  1.  The  TPRI  is  plotted  versus 
the  performance  achieved  using  DD  as  a  parameter.  The  TPRI  has  a  value  of  zero 
when  all  the  performance  measures  have  been  achieved.  The  DD  values  that  are 
between  the  theoretical  bounds  of  zero  and  one  indicate  the  perceived  level  of  difficulty 
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Performance  Risk  Index 


Performance  Achieved 


FIGURE  1.  TPRI  PER  ACHIEVED  PERFORMANCE 
V/ITH  DEGREE  OF  DIFFICULTY  AS  A  PARAMETER 

in  developing  the  technology  to  meet  the  required  system  performance  require- 
ments. 

The  theoretical  lower  boundary  of  TPRI  is  identified  at  DD=0,  and  corresponds 
to  the  line  with  the  lowest  TPRI  value.  This  boundary  indicates  that  there  is  no 
perceived  performance  risk  with  guaranteed  success,  and  it  is  equivalent  to 
Garvey’s  approach  of  unmet  performance  as  the  measure  of  system  performance 
risk.  As  the  DD  increases,  the  TPRI  also  increases.  As  expected,  non-zero  DD 
yields  a  non-linear  behavior  between  the  achieved  performance  and  the  corre¬ 
sponding  TPRI  value.  The  DD=1.0  provides  the  theoretical  upper  bound,  indi¬ 
cating  a  guaranteed  failure.  There  is  a  maximum  TPRI  difference  of  0.18  be¬ 
tween  the  two  theoretical  bounded  cases.  The  TPRI  provides  a  realistic  measure 
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lABU  2,  MEASURED  PERFORAAANCE  OF  TPMS  PER  TIME  PERIOD 


Measured  Performance,  m 

Technical  Performance  Measures 

Hi 

m 

A1  -  Average  Processing  Delay  (msec) 

A2  -  Mean  Time  to  Repair  (minutes) 

A3  -  Payload  Weight  (lbs) 

A4  -  Engagement  Coordination  (sec) 

A5  -  Interceptors  Available  (#  of  units) 

A6  -  Mean  Time  Between  Failure  (hrs) 

A7  -  Single  Shot  Success  Probability  (%) 

A8  -  Damage  Assessment  Accuracy  (%) 

A9  -  Software  Coding  (#  coded  modules) 

1 

10 

950 

0.01 

150 

500 

0.95 

0.995 

763 

3 

50 

2112 

0.1 

67 

100 

0.87 

0.6 

578 

2.86 

43 

1764 

0.04 

128 

189 

0.89 

0.878 

643 

1.18 

43 

1328 

0.032 

134 

223 

0.91 

0.94 

687 

1.09 

27 

1189 

0.02 

139 

348 

0.934 

.0945 

698 

1.03 

12 

1008 

0.01 

142 

379 

0.94 

0.999 

723 

0.98 

9 

948 

0.01 

159 

521 

0.99 

1 

763 

of  performance  risk  associated  with  the  technology  to  meet,  or  exceed,  the 
established  threshold.  Thus,  the  TPRI  yields  valuable  information  to  support 
the  acquisitionist  and  the  technologist  in  making  sound  decisions  regarding 
the  performance  risk  as  a  component  of  the  transition  readiness  of  a  technology. 


ANALYSIS 

The  TPRI  is  applied  to  existing  published  data  (Garvey  &  Cho,  2003)  as  shown 
in  Table  2.  This  data  consists  of  a  set  of  nine  TPMs,  corresponding  threshold  values, 
and  the  measured  performance  over  the  six  various  points  of  time.  An  assessment  of 
the  levels  of  measured  performance  across  each  of  the  TPMs  during  the  time  periods 
is  presented.  For  example,  the  average  processing  delay  increases  in  performance  by 
decreasing  the  average  processing  delay,  measured  in  milliseconds,  from  3  msec  at 
t=l,  to  1 .03  msec  at  t=5,  to  exceeding  the  TPM  threshold  (of  1  msec)  with  a  measured 
performance  of  0.98  msec  at  the  sixth  time  period.  The  measured  performance  data 
contained  in  Table  2  indicates  each  of  the  nine  TPM  thresholds  are  either  met  or 
exceeded  at  the  sixth  time  period. 


Cxottiples  of  this  detreosing  pofformanio  bohavior 
would  ittdude  the  following  four  TPMs:  average 
proressing  delays,  mean  time  to  repair,  payload 
weight,  and  engagement  roordination. 


The  achieved  performances  for  TPMs  with  decreasing  performance  behavior  are 
calculated  using  Equation  1.  Examples  of  this  decreasing  performance  behavior  would 
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TABLE  3.  ACHIEVED  PERFORMANCE  (PERCENTAGE) 
OF  TPMS  PER  TIME  PERIOD  _ 


Achieved  Performance,  A 


Technical  Performance  Measures 


Threshold 


A1  ■  Average  Processing  Delay  (msec) 

1 

0.33 

0.35 

0.85 

0.92 

0.97 

1 

A2  -  Mean  Time  to  Repair  (minutes) 

10 

0.20 

0.23 

0.23 

0.37 

0.83 

1 

A3  -  Payload  Weight  (lbs) 

950 

0.45 

0.54 

0.72 

0.80 

0.94 

1 

A4  -  Engagement  Coordination  (sec) 

0.01 

0.10 

0.25 

0.31 

0.50 

1.00 

1 

A5  -  Interceptors  Available  (#  of  units) 

150 

0.45 

0.85 

0.89 

0.93 

0.95 

1 

A6  -  Mean  Time  Between  Failure  (hrs) 

500 

0.20 

0.38 

0.45 

0.70 

0.76 

1 

A7  -  Single  Shot  Success  Probability  (%) 

0.95 

0.92 

0.94 

0.96 

0.98 

0.99 

1 

A8  -  Damage  Assessment  Accuracy  (%) 

0.995 

0.60 

0.88 

0.94 

0.95 

1.00 

1 

A9  -  Software  Coding  (#  coded  modules) 

763 

0.76 

0.84 

0.90 

0.91 

0.95 

1 

include  the  follcwing  four  TPMs:  average  processing  delays,  mean  time  to  repair, 
payload  weight,  and  engagement  coordination.  In  order  to  meet  the  TPM  threshold, 
the  performance  behavior  is  to  be  minimized. 

In  a  similar  manner.  Equation  2  is  applied  to  calculate  the  achieved  performances 
for  TPMs  with  increasing  performance  behavior.  Examples  of  this  performance 
behavior  would  include;  number  of  interceptors  available,  mean  time  between 
failure,  single  shot  success  probability,  damage  assessment  accuracy,  and  num¬ 
ber  of  software  modules  written.  The  performance  behavior  of  these  TPMs  must 
be  maximized  to  meet  or  exceed  the  TPM  threshold. 

The  resulting  achieved  performance  data  for  each  of  the  nine  TPMs  and  each 
time  measurement  are  tabulated  in  Table  3.  The  achieved  performance  measure 
is  bounded  by  0  and  1.  The  larger  number  value  of  achieved  performance  in¬ 
dicates  that  the  technology  performance  is  approaching  the  TPM  threshold.  An 
achieved  performance  of  1  indicates  that  the  technology  has  either  met  or  ex¬ 
ceeded  the  TPM  threshold  and  has  accomplished  the  goal  of  entering  the  accept¬ 
able  risk  region. 

TaWe  4  provides  the  listing  of  the  nine  TPMs  and  the  associated  achieved 
performance.  The  Degree  of  Difficulties  are  arbitrarily  selected  (for  demonstration 
purposes)  for  each  TPM  per  each  of  the  six  time  periods.  The  TPRI  vras  computed 
for  each  individual  TPM  requirement  for  each  time  period.  For  example,  at  time  period 
t=l,  the  average  processing  delay  TPM  has  a  TPRI  of  0.75,  0.74  at  t=2,  0.19  at  t=3, 
and  continues  to  improve  imtil  t=6,  when  the  computed  TPRI  is  zero,  indicating  that 
the  TPM  was  met  or  exceeded. 

The  total  system  level  TPRI  is  also  calculated  for  the  technology  for  each  of  the 
six  time  periods.  The  aggregate  TPRI  for  the  technology  is  computed  by  averaging 
the  TPRI  values  of  the  nine  individual  TPM  requirements,  and  is  identified  in  Table 
4.  As  the  technology  advances  in  improved  performance,  the  TPRI  value  decreases, 
indicating  lower  system  performance  risks.  For  example,  the  aggr^ate  TPRI  for  the 
technology  is  0.64  for  the  first  time  period,  0.49  for  the  second  time  period,  and 
continues  to  improve  (decrease)  through  the  sixth  time  period,  when  the  aggregate 
TPRI  is  zero. 

Figure  2  depicts  the  aggregate  technology-level  TPRI  value  for  each  time 
period.  Observation  of  this  figure  indicates  the  areas  of  significant  risk  and  helps 
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TABLE  4A.  TPMS  AND  ASSOCIATED  ACHIEVED  PERFORMANCE 


Technical  Performance  Measures^ 

A1  -  Average  Processing  Delay  (msec) 
A2  -  Mean  Time  to  Repair  (minutes) 

A3  -  Payload  Weight  (lbs) 

A4  -  Engagement  Coordination  (sec) 

A5  -  Interceptors  Available  (#  of  units) 

A6  -  Mean  Time  Betv/een  Failure  (hrs) 
A7  -  Single  Shot  Success  Probability  (%) 
A8  -  Damage  Assessment  Accuracy  (%) 
A9  •  Software  Coding  (#  coded  modules) 


Achieved 

Performance 

0.33 

0.20 

0.45 

0.10 

0.45 

0.20 

0.92 

0.60 

0.76 


Degree  of 
Difficulty 


TPRI  per 
Requirement 


TotalTPRI,att1 

=  0.64 

Twhnical  Performance  Measures 

Achieved 

Performance 

Degree  of 
Difficulty 

TPRI  per 
Requirement 

A1  -  Average  Processing  Delay  (msec) 

0.35 

0.5 

0.74 

A2  -  Mean  Time  to  Repair  (minutes) 

0.23 

0.9 

0.86 

A3  -  Payload  Weight  (lbs) 

0.54 

0.5 

0.56 

A4  -  Engagement  Coordination  (sec) 

0.25 

0.5 

0.82 

A5  -  Interceptors  Available  (#  of  units) 

0.85 

0.3 

0.18 

A6  -  Mean  Time  Between  Failure  (hrs) 

0.38 

0.9 

0.76 

A7  -  Single  Shot  Success  Probability  (%) 

0.94 

0.3 

0.08 

AS  -  Damage  Assessment  Accuracy  (%) 

0.88 

0.5 

0.17 

A9  -  Software  Coding  (#  coded  modules) 

0.84 

0.5 

0.22 

=  0.49 

Technical  Performance  Measures 

Achieved 

Degree  of 

TPRI  per 
Requirement 

Performance 

Difficulty  1 

A1  -  Average  Processing  Delay  (msec) 

0.85 

0.3 

0.19 

A2  -  Mean  Time  to  Repair  (minutes) 

0.23 

0.9 

0.86 

A3  -  Payload  Weight  (lbs) 

0.72 

0.3 

0.34 

A4  -  Engagement  Coordination  (sec) 

0.31 

0.5 

0.77 

A5  -  Interceptors  Available  (#  of  units) 

0.89 

0.3 

0.13 

A6  -  Mean  Time  Between  Failure  (hrs) 

0.45 

1.9 

0.70 

A7  -  Single  Shot  Success  Probability  (%) 

0.96 

1.3 

0.05 

A8  -  Damage  Assessment  Accuracy  (%) 

0.94 

0.3 

0.07 

A9  -  Software  Coding  (#  coded  modules) 

0.90 

0.3 

0.13 

Total  TPRi,  at  t3 

=  0.36 

PFRFORMANCE  ^BASED 


TABLE  4A  TPMS  AND  ASSOCIATED  ACHIEVED  PERFORMANCE  (cont) 


Technical  Performance  Measures 

A1  -  Average  Processing  Delay  (msec) 
A2  -  Mean  Time  to  Repair  (minutes) 

A3  -  Payload  Weight  (lbs) 

A4  -  Engagement  Coordination  (sec) 

A5  -  Interceptors  Available  (#  of  units) 

A6  -  Mean  Time  Between  Failure  (hrs) 

A7  -  Single  Shot  Success  Probability  (%) 
A8  -  Damage  Assessment  Accuracy  (%) 
A9  -  Software  Coding  (#  coded  modules) 


JTechnjcal  Performance  Measures 

At  -  Average  Processing  Deiay  (msec) 
A2  -  Mean  Time  to  Repair  (minutes) 
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decision  makers  with  providing  additional  information  pertaining  to  the  perfor¬ 
mance  risk  involved  with  a  technology  to  meet  the  threshold  values  associated 
with  TPMs.  At  the  sixth  time  period,  the  technology  has  achieved  the  acceptable 
risk  region  and  has  a  zero-valued  TPRI. 


TPRI  BENEFITS 

The  TPRI  provides  a  realistic  performance  risk  assessment  based  on  performance 
requirements,  Degree  of  Difficulty,  and  unmet  performance.  These  components  are 
combined  in  a  closed-loop  feedback  manner  to  calculate  the  technology  performance 
risk.  This  decision  support  tool  provides  insight  to  the  risk  involved  in  the  unmet 
performance  to  meet  TPM  thresholds,  and  the  level  of  activity  required  to  meet  or 
exceed  the  threshold.  TPRI  applies  performance  risks  associated  with  an  unmet  re¬ 
quirement  as  correction/feedback  to  the  achieved  performance.  Since  TPRI  is  based 
on  meeting  TPM  thresholds  and  identifying  associated  risks  with  unmet  requirement, 
it  provides  common  pound  between  technologist  and  acquisitionist.  As  a  result,  ari 
improved  understanding  of  the  technology’s  capabilities  to  support  the  acquisition 
system  is  gained. 


CONCLUSION 

This  research  has  focused  on  the  development  and  demonstration  of  a  quantitative 
methodology  to  evaluate  and  monitor  technology  to  determine  transition  readiness. 
The  TPRI  provides  a  means  to  assess  potential  technologies  and  assist  the  decision 
maker  in  where  to  apply  resources  to  address  unmet  requirements.  The  TPRI  supports 
monitoring  of  performance  of  a  technology  against  threshold  limits.  It  integrates 
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the  technology  degree  of  difficulty  along  with  the  system  unmet  requirements 
in  a  closed  loop  to  gain  additional  information  regarding  a  technology’s  perfor¬ 
mance  risk  over  time.  Additionally,  it  reduces  the  probability  associated  with 
immature  technology  being  transitioned  to  a  weapon  system  prematurely.  This 
approach  is  anticipated  to  contribute  toward  level  of  success  pertaining  to  the 
integration  of  technology  into  system(s)  of  the  aerospace  and  defense  domains. 


NEXT  STEPS 

Future  efforts  will  include  examining  methods  to  combine  this  quantitative 
Technology  Performance  Risk  Index  (TPRI)  with  a  technology  maturity  metric, 
such  as  the  widely  utilized  Technology  Readiness  Level  (TRL).  Various  racing 
and  weighting  schemes  are  planned  to  be  examined  for  potential  applicability  in 
the  TPRI  calculation.  There  are  plans  to  apply  the  TPRI  as  a  decision  support 
tool  to  assist  a  decision  maker  with  evaluating  and  selecting  the  best  of  identi¬ 
fied  multiple  technologies  across  same  set  of  TPMs. 
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SERVICE  LEVEL  AGREEMENTS 
AS  VEHICLES  FOR 
MANAGING  ACQUISITION 
OF  SOFTWARE-INTENSIVE 

SYSTEMS 

CDR  LEONARD  T.  GAINES,  USN  AND 
DR.  JAMES  BRET  MICHAEL 


Service  level  agreements  (SLAs)  can  be  used  as  a  means  to  manage  the 
acquisition  of  software- intensive  systems.  The  SLAs  support  performance- 
based  acquisition  by  stating  in  measurable  terms  the  service  to  be  per¬ 
formed,  the  level  of  service  that  is  acceptable,  the  way  in  which  the  service 
level  is  to  be  measured,  and  the  incentives  for  the  provider  of  information 
technology  products  and  services  to  meet  the  agreed-to  target  levels  of 
quality.  The  SLAs  are  traditionally  used  in  outsourcing  contracts  for  post¬ 
production  support.  This  article  proposes  a  new  approach  by  using  SLAs  in 
software  acquisition  to  support  quality  and  process  control  throughout  the 
entire  lifecycle  of  a  software -intensive  system.  This  article  defines  SLAs, 
discusses  software  quality,  and  describes  how  SLAs  can  be  utilized  to  incor¬ 
porate  requirements  pertaining  to  product,  process,  project,  and  deploy¬ 
ment  quality  throughout  the  software  lifecycle. 


AS  advances  in  information  technology  (IT)  increase  worker  productivity  and 
encourage  the  adoption  of  new  ways  of  conducting  business,  organizations 
and  end  users  become  ever  increasingly  dependent  upon  that  technology. 
Consequently,  the  strategic  and  tactical  advantages  that  IT  affords  an  organization 
places  pressure  on  the  organization’s  IT  department  to  provide  quality  services  and 
products.  Intermptions  to  software-intensive  systems  are  having  a  far  greater  impact 
than  before  in  terms  of  opportunity  loss,  revenue  loss,  customer  dissatisfaction. 
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and  inefficiency.  As  managers  realize  that  their  mission-critical  processes  are  tied 
to  IT  services  and  products,  they  are  demanding  correspondingly  greater  levels 
of  both  performance  and  quality  in  these  services  and  products,  despite  the  fact 
that  acquiring  and  maintaining  software-intensive  systems  is  increasingly  chal¬ 
lenging  from  both  a  managerial  and  technical  perspective. 

In  addition,  an  organization  may  not  have  enough  personnel  with  the  right 
mix  of  IT  skills,  knowledge,  and  experience  within  their  organization  to 
provide  high-enough  quality  IT  products  and  services  for  the  organization’s 
employees  and  business  partners.  Rather  than  hire  IT  specialists  or  invest  in 
training  for  its  internal  staff,  the  organization  may  have  the  option  to  outsource 
to  external  service  providers  (ESPs)  their  IT  services  and  products.  This  strategy 
has  been  followed  by  the  US.  Department  of  Defense  (DoD)  in  procuring  large 
software-intensive  systems  such  as  the  well-known  Navy/Marine  Corps  Intranet 
(NMCI)  and  Ballistic  Missile  Defense  System  (BMDS).  The  hope  is  that  ESPs 
can  readily  supply  different  mixes  of  IT  specialists  to  meet  the  ever-changing 
requirements  an  organization  has  for  IT  products  and  services,  at  a  lower  cost 
than  enlarging,  shrinking,  or  retraining  its  internal  IT  staff.  In  addition  to  pro¬ 
viding  access  to  cutting-edge  technology  and  skilled  staff,  ESPs  share  the  project 
risk  and  make  it  possible  for  organizations  to  concentrate  on  core  competencies 
(King,  2001). 


Reiian<e  on  outsouning  adds  another  layer 
of  {omplexity  onto  the  management 
of  information  systems. 


Reliance  on  outsourcing  adds  another  layer  of  complexity  onto  the  management  of 
information  systems.  Outsourcing  efforts  require  additional  discipline  and  management 
oversight  that  may  not  be  necessary  with  in-house  development  or  maintenance  of 
information  systems.  Outsourcing  requires  skill  in  software  acquisition  as  well  as 
project  management.  Program  managers  need  to  be  accomplished  in  software- 
requirements  engineering,  software  development,  the  contracting  process,  requirement- 
change  management,  contract  management  and  oversight,  and  contractor-relationship 
management. 

When  outsourcing  software-intensive  programs,  the  program  manager  must 
be  explicit  in  stating  the  services  to  be  performed,  in  addition  to  identifying  the 
metrics  and  means  to  determine  whether  the  contractor  has  satisfied  those  re¬ 
quirements.  However,  it  is  challenging  to  develop,  manage,  and  enforce  a  con¬ 
tract  for  software  services  that  will  ensure  that  the  contractor  delivers  the  speci¬ 
fied  services  or  end  product  within  prescribed  levels  of  performance  and  quality. 
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Managing  software-intensive  information  systems  can  also  be  challenging. 
Utilizing  the  latest  technology  to  exploit  information  requires  highly  developed 
intellectual  and  managerial  skills.  The  difficulty  in  managing  these  systems  has 
been  demonstrated  by  the  numerous  system  development  and  maintenance 
projects  within  the  US.  Department  of  Defense.  Many  of  these  projects  lacked 
sound  planning,  had  inadequate  controls,  were  without  effective  measurements 
for  success,  and  failed  to  meet  user  expectations  (U.S.  General  Accounting  Office 
[GAO],  2001;  Department  of  Defense,  Office  of  the  Inspector  General  [DoD 
OIG],  2000).  However,  the  private  sector  also  has  problems  managing  IT  projects. 
The  2003  Standish  Group’s  CHAOS  research  report  indicates  that  of  the  13,522 
IT  projects  studied;  only  34  percent  of  projects  were  considered  a  success,  and 
only  52  percent  of  required  features  were  incorporated  into  the  released  product 
(Standish  Group,  2003). 

Despite  software's  imreused  importanre 
to  organizations,  the  quality  of  software 
is  often  lathing. 

Despite  software’s  increased  importance  to  organizations,  the  quality  of  software 
is  often  lacking.  Some  would  aigue,  such  as  Mann  (2002),  that  despite  the  advances 
in  software  engineering  theory,  processes,  methodology,  and  tools,  software  quality 
is  actually  getting  worse.  Mann’s  reasons  center  around  four  main  points:  1)  developers 
are  rushing  software  to  market,  2)  software  is  poorly  designed,  3)  developers  and 
testers  lack  the  requisite  skills,  and  4)  programs  are  not  managed  well.  Given  the 
difficulties  in  developing  software-intensive  systems,  performance  and  quality 
requirements  should  be  well  defined  and  monitored  throughout  the  software’s  lifecycle. 
In  this  paper  we  explain  how  service  level  agreements  (SLAs)  can  be  utilized  in 
software  acquisition  to  improve  the  quality  and  management  of  software  throughout 
its  lifecycle. 

SLAs  are  a  means  of  incorporating  Performance-based  Service  Contracting  (PBSC) 
into  software  acquisition.  SLAs  are  similar  to  a  Quality  Assurance  Plan  (QAP) 
in  that  they  define  quality  and  performance  requirements,  they  contain  penalties  for 
non-performance,  and  they  establish  quality  control  methods  to  ensure  compliance. 


METHODS 

In  this  article  we  identify  areas  throughout  the  lifecycle  (requirements  genera¬ 
tion  through  post-production  support)  of  software-intensive  systems  in  which 
SLAs  can  be  used  to  manage  the  contractual  provisions  of  insourced  or  outsourced 


IT  services  and  products.  In  this  section  we  define  terms  associated  with  SLAs  and 
quality,  in  addition  to  describing  how  SLAs  can  be  utilized  in  requirements  engi¬ 
neering,  design,  post-production  support,  and  program  management  to  achieve 
target  levels  of  performance  and  quality  for  IT  services  and  products. 

An  SLA  is  a  contractual  mechanism  between  a  provider  of  services  and  a  cus¬ 
tomer  that  defines  a  level  of  performance  (Sturm,  Morris,  &  Jander,  2000).  This 
agreement  defines  in  measurable  tenns  the  service  to  be  performed,  the  level  of 
service  that  is  acceptable,  the  means  to  determine  if  the  service  is  being  provided 
at  the  agreed  upon  levels,  and  incentives  for  meeting  agreed  upon  service  levels. 
SLAs  contain  quality-of-service  (QoS)  requirements,  including  how  QoS  is  to  be 
measured.  SLAs  also  set  forth  the  roles  and  responsibilities — in  contractual  form — 
of  both  the  organization  and  the  provider  of  the  services  and  products. 


••.Ever-growing  reliante  on  software-intensive 
systems  for  fondutting  business  and  retent 
advantes  in  both  tethniques  and  tools  for 
spetifying  SLAs  and  monitoring  tompliante  to  SLAs 
have  all  tontributed  to  the  rise  in  the  atteptante 
of  SIAs  within  the  pubiit  and  private  setters. 


SLAs  ^e  not  a  new  concept:  they  have  been  around  since  the  1960s.  However, 
ever-growing  reliance  on  software-intensive  systems  for  conducting  business  and  recent 
advances  in  both  techniques  and  tools  for  specifying  SLAs  and  monitoring  compliance 
to  SLAs  have  all  contributed  to  the  rise  in  the  acceptance  of  SLAs  within  the  puWic 
and  private  sectors.  If  SLAs  are  not  included  in  a  system-acquisition  contract,  the 
acquiring  oiganization  has  little  leverage  over  the  provider  of  the  IT  services  or  prod¬ 
ucts  to  ensure  that  the  oiganization^  performance  and  QoS  requirements  are  met. 

There  is  a  subtle  distinction  between  an  SLA  and  a  traditional  contract  requirement. 
SLAs  are  also  requirements,  but  they  are  different  contractually:  SLAs  contain  more 
detail,  describe  incentives  for  meeting  thresholds,  and  specify  incentives  for  meeting 
performance  and  QoS  requirements.  When  SLAs  are  used,  the  contracting  officer  can 
withhold  incentive  p^ents  or  levy  penalties  if  quality  levels  are  not  met  during  the 
time  period  stated  in  the  SLA  (usually  a  month  or  a  quarter).  Requirements  are  gen¬ 
erally  only  measured  at  the  end  of  a  milestone  decision.  In  most  contracts,  the  only 
recourse  if  a  requirement  is  not  met  is  to  cancel  the  contract,  terminate  any  ongoing 
contactor  support,  or  give  the  contactor  a  poor  performance  rating.  Terminating  a 
project  can  be  difficult,  especially  if  the  software-intensive  system  to  be  acquired  is 
mission-critical.  SLAs  specify  the  degree  of  recourse  if  a  requirement  is  not  met. 
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One  of  the  ways  that  SLAs  can  help  improve  software  quahty  is  that  they  focus  the 
attention  of  the  acquisition  professional  on  the  nonfunctional  requirements  (e.g.,  reli¬ 
ability,  security,  testability)  that  otherwise  might  be  ignored  until  the  test  phase  in  the 
absence  of  SLAs  (Bass,  Clements,  &  Kazman,  1998).  SLAs  contractually  mandate  the 
performance  and  quality  attributes  that  users,  program  managers,  and  software  engi¬ 
neers  consider  essential  for  a  system  to  support  the  underlying  business  process.  They 
help  make  explicit  many  of  the  quality  fectors  that  users  may  implicitly  assume.  SLAs 
also  specify  the  quality  metrics  by  which  the  software  quality  requirements  are  mea¬ 
sured.  Measuring  and  monitoring  performance  and  quality  of  services  and  products 
makes  it  possible  for  an  organization  to  judge  whether  performance  and  quality 
requirements  have  been  met.  Measurements  also  support  early  detection  and  resolution 
of  problems  with  quality. 


Quality  tan  be  viewed  from  numerous 
perspettives,  and  tertain  attributes  are  more 
preferable  to  others  depending  on  tbe  mission  the 
software-intensive  system  is  supposed  to  support 


Before  Hismssing  SLAs  and  how  they  can  improve  the  performance  and  quality 
in  the  various  phases  of  a  software-intensive  system’s  lifecycle,  we  first  define  the 
term  software  quality.  There  are  numerous  definitions  of  quality.  The  International 
Standards  Organization  (ISO)  standard  9126  defines  software  quality  as  the  total¬ 
ity  of  features  and  characteristics  of  a  software  product  that  bear  on  its  ability  to 
satisfy  stated  or  implied  needs.  Another  definition  is  that  software  quality  is  con¬ 
formance  to  explicitly  stated  functional  and  performance  requirements,  explicitly 
documented  development  standards,  and  implicit  characteristics  that  are  expected 
of  all  professionally  developed  software  (Pressman,  2001).  Others  believe  that  an 
IT  system  will  not  be  perceived  as  a  quality  product  if  the  product  does  not  per¬ 
form  according  to  the  end-user’s  perspective  (Wiegers,  2002).  The  Institute  of 
Electrical  and  Electronics  Engineers  (IEEE)  standard  610-1990  does  incorporate 
user  needs:  it  defines  software  quality  as  the  degree  to  which  a  system,  compo¬ 
nent,  or  process  meets  specified  requirements  and  meets  customer  or  user  needs 
and  expectations  (Schmidt,  2000). 

In  this  article  we  adopt  the  IEEE  definition  and  specialize  it  to  encompass  four 
specific  areas  of  focus:  1)  product  quality,  2)  project  quality,  3)  process  quality, 
and  4)  post-production  quality.  Each  of  the  areas  has  quality  attributes  associated 
with  it  that  describe  the  degree  to  which  the  software  possesses  certain  characteris¬ 
tics.  SLAs  utilize  quality  metrics  and  threshold  levels  to  assign  quantitative  measure¬ 
ments  to  the  quality  attributes.  The  quality  metrics  are  then  monitored  to  ensure 
compliance. 
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Quality  can  be  viewed  from  numerous  perspectives,  and  certain  attributes  are  more 
preferable  to  others  depending  on  the  mission  the  software-intensive  system  is 
supposed  to  support.  Numerous  quality  attributes  and  quality  models  have  been 
developed  to  measure  or  predict  software  quality.  Selecting  the  appropriate  quality 
attributes  for  the  project  is  the  responsibility  of  the  SLA  development  team,  the 
program  manager,  and  stakeholders. 

PRODUCT  QUALITY 

Ifroduct  quality  is  concerned  with  the  relationships  between  the  characteristics  and 
attributes  of  a  product  and  its  ability  to  satisfy  stated  requirements  and  implicit  needs; 
this  area  can  also  be  referred  to  as  end-product  quality.  In  addition  to  the  functional 
aspects  of  a  system,  the  end  users  want  the  product  to  exhibit  certain  quality 
characteristics  that  will  assist  them  in  performing  their  task.  From  a  user%  perspective, 
some  of  the  common  quality  attributes  include  availability,  usability,  integrity, 
interoperability,  and  reliability.  Personnel  involved  in  the  development  of  software  or 
its  maintenance  may  be  more  concerned  with  the  software  attributes  such  as  portability 
testability,  maintainability,  and  reusability  (Wiegers,  2002). 

The  ISO  standard  9126-1  quality  model  imorporates 
the  following  quality  fa€tors:  fundionality,  reliability, 
usability,  effkienty,  maintainability,  and  portability 


There  are  numerous  quality  models  that  can  be  incorporated  into  SLAs  that  assign 
quantitative  measurements  to  various  product  quality  attributes.  Some  of  the  better- 
known  quality  models  include  those  proposed  by  McCabe  in  ^^^lich  software  complexity 
is  a  function  of  the  number  of  conditional  statements  in  the  code,  and  those  of  Halstead 
in  which  software  complexity  is  a  function  of  the  number  of  operators  and  operands 
(Ogasawara,  Yamada,  &  Kojo,  1996).  The  ISO  standard  9126-1  quality  model 
incoiporates  the  following  quality  fectors:  functionality,  reliability,  usability,  efficiency, 
maintainability,  and  portability  (Cross,  2002).  There  are  also  numerous  software  qual¬ 
ity  models  that  concentrate  on  specific  quality  fectors  such  as  complexity.  Zuse  (1991) 
identifies  over  90  models  for  describing  software  complexity.  Other  quality  models  are 
specific  to  object-oriented  systems  (Coppick  &  Cheatham,  1992),  specific  languages 
(Pritchett,  2001),  commercial  off-the-shelf  (COTS)  components  (Bertoa  &  Vhllecillo, 
2002),  and  others,  are  only  applicable  at  run-time  (Bass  et  al.,  1998). 

PROJEa  QUALITY 

Project  quality  is  concerned  with  metrics  that  allow  an  organization  to  manage, 
track,  and  improve  the  quality  of  the  software  development  effort;  this  area  could 
be  viewed  as  in-process  quality  management  (Hilburn  &  Townhidnejad,  2000). 


Some  of  the  common  project  quality  metrics  that  Motorola  used  to  measure  its 
software  development  projects  included  software  defect  density,  adherence  to 
schedule,  estimation  accuracy,  reliability,  requirements  tracking,  and  fault-type 
tracking  (Daskalantonakis,  1992).  Another  metric  used  in  assessing  project  quality 
is  risk,  \!vhich  can  be  defined  as  ai^  variable  within  a  project  that  results  in  project 
failure.  General  risk  areas  are  schedule  risk,  requirements  risk,  budget  risk,  and 
personnel  risk  (Padayachee,  2002).  There  are  a  number  of  risk-assessment  models 
including  Gilb’s  (1993)  risk  heuristics,  Boehm’s  (1991)  classification  of  risk,  Noguiera 
de  Leon’s  (2000)  risk  assessment  model,  Keil’s  identification  of  risk  fectors  (Keil, 
Cule,  Lyytinen,  &  Schmidt,  1998),  and  risk  models  associated  with  enterprise 
software  projects  (Sumner,  2000). 

PROCESS  QUALITY 

Process  quality  is  concerned  with  the  processes,  planning,  and  controls  used  to 
develop  and  manage  the  software  product.  It  could  be  argued  that  the  quality  of 
the  development  process  is  the  best  predictor  of  software  product  quality  (Fenton, 
Krause,  &  Neil,  2002).  Repeatable  software  processes  such  as  the  Software  En¬ 
gineering  Institute’s  Capability  Maturity  Model  for  software  (SW-CMM  also  known 
as  CMM),  which  lists  five  levels  of  organizational  maturity,  and  the  ISO  9001,  are 
designed  to  improve  software  quality,  productivity,  predictability,  and  time  to  market 
(McGuire,  1996).  There  is  empirical  evidence  that  supports  the  relationship  be¬ 
tween  process  maturity  and  software  quality  (Harter  &  Slaughter,  2000). 


Process  quality  is  tonterned  with  the  professes, 
planning,  and  lontrols  used  to  develop  and 
manage  the  software  produrt. 


Other  models  of  process  quality  include  the  Software  Engineering  Institute’s  new 
Capability  Maturity  Model  Integration  (CMMI).  CMMI  integrates  three  CMM  models 
into  one  to  eliminate  problems  with  different  architecture,  semantics,  and  approaches. 
Humpheiy  (1996)  also  developed  a  process  model  called  the  I^rsonal  Software  Process 
(PSP)  to  assist  software  engineers  in  producing  quality  software.  Other  process  models 
include  clearaoom  engineering  that  has  shown  reduced  errors  per  KLOC  for  small 
projects  (Fenton  &  Neil,  1999),  and  the  quality  management  metric  (QMM) 
(Osmundson,  Michael,  Machniak,  &  Grossman,  2003).  There  are  also  numerous 
IEEE  and  ISO  standards  that  provide  processes  on  everything  from  software  en¬ 
gineering  product  evaluation  (e.g.,  ISO/IEC  14598)  to  selecting  appropriate  qual¬ 
ity  metrics  (e.g.,  IEEE  Std.  1061-1998). 
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POST-PRODUCTION  QUALITY 


The  last  area  of  focus  is  on  post-production  quality  or  deployed-application 
management.  Many  of  the  quality  models  involving  deployed  applications  are 
concerned  with  software  maintenance  and  the  quality  factors  that  make  maintenance 
cost-effective.  Some  of  the  maintenance  quality  factors  deal  with  ease  of  change 
(Royce,  1990),  architectural  design  to  promote  maintenance  (Garlan,  2000),  defect 
management  (Kajla>Mattsson,  1998),  organizational  structure  (Briand,  Melo,  Seaman, 
&  Basili,  1995),  and  change  management  (Bennett  &  Rajlich,  2000). 

Quality  does  not  only  apply  to  the  application  itself,  it  is  also  concerned  with  the 
IT  system  as  a  whole,  across  distributed  components.  Part  of  that  distributed  system 
is  the  network.  There  are  numerous  quality  metrics  that  can  be  applied  to  network 
QoS  (Moeller  et  al.,  2000).  Quality  metrics  are  also  applied  to  the  host  server.  Quality 
metrics  such  as  application  resource  utilization  (Aries,  Baneijee,  Brittan,  Dillon, 
Kowalik,  &  Lixvar,  2002),  bandwidth  utilization  (Eager,  Vernon,  &  Zahoijan,  2001), 
concurrent  user  management  (Aweya,  Ouellette,  Montuno,  Doray,  &  Felske,  2002), 
and  server  performance  (Gama,  Meira,  Carvalho,  Guedes,  &  Almeida,  2001)  are 
also  utilized  to  address  system-level  quality.  Other  areas  where  quality  metrics  are 
applied  include  total  cost  of  operation  (TCO),  help  desk  support  metrics,  backups, 
storage,  configuration  management,  and  security. 


When  developing  ihe  SlAs,  it  is  essential  that 
all  stakeholders  he  represented  in  this  effort, 
as  the  system  requirements  need  to  represent 
all  of  their  needs  for  quaiity. 


There  are  numerous  software  quality  models  and  metrics  that  can  be  incoiporated 
into  SLAs.  The  models  or  quality  factors  chosen  will  depend  upon  those  quality 
attributes  that  best  support  the  underlying  business  processes  of  the  organization 
acquiring  the  software-intensive  system.  SLAs  should  only  incorporate  software 
metrics  that  are  meaningful,  quantitative,  and  measurable. 

When  developing  the  SLAs,  it  is  essential  that  all  stakeholders  be  represented 
in  this  effort,  as  the  system  requirements  need  to  represent  all  of  their  needs  for 
quality.  The  development  team,  consisting  of  representation  for  all  stakeholders, 
must  resolve  a  number  of  issues.  These  include,  for  instance,  identifying  quality 
requirements,  determining  the  various  levels  of  service  that  are  needed,  detailing 
quality  metrics  that  are  meaningful  and  measurable,  assessing  risk,  resolving  con¬ 
flicting  quality  requirements,  and  prioritizing  the  quality  requirements.  When  quo¬ 
tations  are  received  from  vendors  (i.e.,  the  providers  of  IT  services  and  products). 
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the  group  also  needs  to  perform  a  cost-benefit  analysis  based  on  the  levels  of  quality 
thresholds  desired  and  the  budget  allotted  for  the  acquisition.  The  group  develop¬ 
ment  of  SLAs  helps  the  various  stakeholders  understand  each  others’  biases,  view¬ 
points,  concerns,  terminology,  and  perceptions — ^that  understanding  is  essential  in 
requirements  engineering. 


REQUIREMENTS  ENGINEERING 

Requirements  engineering  provides  the  building  blocks  for  all  other  efforts  in  the 
software-engineering  process,  so  if  quality  is  not  addressed  in  the  initial  require¬ 
ments  analysis,  it  is  usually  addressed  at  the  end  of  the  project  in  the  form  of  testing. 
If  quality  requirements  are  implemented  too  late,  the  architecture  that  was  already 
developed  will  dictate  the  solution  space  for  addressing  quality  problems  that  are 
discovered,  or  the  acquisition  authority  will  need  to  approve  major  contract  modi¬ 
fications  to  permit  changes  to  the  requirements,  architecture,  and  other  high-level 
system  artifacts. 


Requirements  engineering  provides  the  buiiding 
bioiks  for  nit  other  efforts  in  the  software¬ 
engineering  protess,  so  if  quality  is  not 
addressed  in  the  initial  requirements  analysis, 
it  is  usually  addressed  at  the  end  of 
the  proiert  in  the  form  of  testing. 


The  process  of  developing  the  SLAs  is  part  of  the  larger  requirement-engineering 
process  and  fosters  discussion  of  the  following:  the  goals  of  the  system  that  must 
be  met,  the  processes  and  tasks  that  the  system  must  perform  to  meet  those  goals, 
and  any  operational  and  organizational  needs,  policies,  standards,  and  constraints. 
Discussions  necessary  to  develop  the  SLAs  will  generate  information  about  the 
application  domain,  business  and  organization  processes/culture,  and  the  intended 
operating  environment  that  the  system  will  be  placed  in.  The  discussions  will  help 
the  requirements  engineer  capture  tacit  knowledge,  identify  constraints,  prioritize 
requirements,  and  justify  how  quality  factors  support  business  needs.  Additionally, 
SLAs  require  quantifiable  quality  metrics,  so  those  quality  requirements  that  cannot 
be  measured,  do  not  provide  value,  or  are  not  realistic  should  not  be  accepted  by 
the  SLA  development  team  or  the  requirements  engineer. 
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In  many  organizations,  senior  management’s  involvement  in  requirements 
engineering  can  be  low  (Bubenko,  1995),  resulting  in  requirements  that  m^  not  be 
related  to  business  visions  and  objectives.  SLAs  help  mitigate  this  problem  because 
they  define  specific  performance  parameters  that  are  tied  to  business  processes.  As 
such,  every  SLA  performance  requirement  must  be  analyzed,  and  validated  to  en¬ 
sure  that  it  is  meaningful,  cost-effective,  and  that  they  add  to  improving  overall 
performance.  Senior  management  m^  also  take  more  notice  when  there  are  con¬ 
tractual  repercussions  (i.e.,  penalties  or  rewards)  associated  with  the  requirements. 


Iiuorponting  standards  in  SLAs  provides  a  tommon 
methodology  that  makes  management  easier  for 
program  managers  and  iontratting  officers  as  they 
provide  the  basis  against  which  activities  can  be 
measured  and  evaluated  (Horch,  1996). 


By  defining  meaningful  and  measurable  metrics  in  the  SLAs,  the  end-users, 
business  managers  and  software  engineers  have  realistic  quantifiable  requirements 
that  can  be  used  to  develop  architectures,  designs,  and  other  software  artifacts. 


DEVELOPMENT 

The  real  contribution  of  SLAs  in  obtaining  quality  software  is  that  the  quality 
factors  addressed  in  the  SLAs  drive  significant  architectural  and  design  decisions. 
If  developers  know  which  of  the  characteristics  are  most  critical  to  project  success, 
they  can  select  the  architecture,  design,  programming,  and  verification  and  valida¬ 
tion  approaches  that  best  achieve  the  specified  quality  goals.  When  customer  re¬ 
quirements  have  been  collected  and  specified,  architecting  and  designing  translates 
the  requirements  into  a  blueprint  that  programmers  can  use  to  build  the  product. 
That  design  can  be  evaluated  and  monitored  to  ensure  quality  factors  are  adequately 
addressed.  In  this  way,  quality  is  designed  in  the  beginning  phases  of  the  lifecycle 
where  it  is  most  cost-effective  to  do  so. 

The  architecture  of  a  software  system  models  its  structure  and  behavior  (Shaw  & 
Garlan,  1996).  Architecture  also  shows  the  correspondence  between  the  system 
requirements  and  the  elements  of  the  constructed  system;  so  by  analyzing  the 
architecture  it  is  possible  to  predict  the  quality  of  a  product  before  it  has  been  built. 
There  are  a  number  of  methods  to  analyze  architectures  (Bass  et  al.,  1998).  How¬ 
ever,  these  methods  do  not  produce  quantifiable  quality  measurements  (Dobrica  & 
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Niemela,  2002),  although  they  can  provide  an  estimate  of  how  well  the  design  will 
satisfy  a  particular  quality  factor. 

SLAs  also  improve  software  quality  in  the  development  phase  by  contractually 
mandating  that  certain  quality-control  measures  (e.g.,  adhering  to  specified  stan¬ 
dards  and  processes)  be  performed.  There  are  numerous  industry-approved  stan¬ 
dards  that  can  be  incorporated  into  SLAs.  Incorporating  standards  in  SLAs  pro¬ 
vides  a  common  methodology  that  makes  management  easier  for  program  man¬ 
agers  and  contracting  officers  as  they  provide  the  basis  against  which  activities 
can  be  measured  and  evaluated  (Horch,  1996).  Standards  can  be  applied  to  all 
aspects  of  development,  maintenance,  and  operation  of  an  information  system. 

Development  processes  can  be  specified  in  an  SLA.  Specifying  specific  pro¬ 
cesses  has  many  of  the  same  advantages  of  specifying  compliance  to  standards. 
Applying  well  defined,  standardized  software-development  processes  can  increase 
software  quality  and  make  the  development  effort  more  cost  effective  and  predict¬ 
able  (Gnatz,  Marschall,  Popp,  Rausch,  &  Schwerin,  2003).  Specifying  the  pro¬ 
cesses  in  the  SLA  helps  to  ensure  that  they  are  recognized  and  adhered  to. 


Applying  well  defined,  standardized  software- 
development  professes  fan  infrease  software 
quality  and  make  the  development  effort  more  fost 
effeUive  and  prediftable  (Gnatz,  Marsfahll,  Popp, 
Rausfh,  &  Sfhwerin,  2003), 


SLAs  also  assist  the  testing  effort  by  identifying  business-critical  processes,  defining 
quantitative  metrics  to  measure  quality  factors,  identifying  testing  procedures,  and 
ensuring  testing  is  conducted  throughout  the  system’s  lifecycle.  SLAs  can  ensure 
that  other  audits  such  as  documentation  reviews,  requirements  reviews,  design  re¬ 
views,  test  plan  reviews,  user  documentation  reviews,  and  implementation  reviews 
are  conducted  (Horch,  1996). 


PROGRAM  MANAGEMENT 

Program  managers  have  to  ensure  that  quality  considerations  are  addressed  early 
in  the  lifecycle  and  they  must  provide  the  proper  amount  of  oversight  to  ensure 
those  quality  factors  are  incorporated  into  the  final  product.  SLAs  can  assist  program 
managers  in  many  of  the  tasks  necessary  to  ensure  quality  is  delivered  in  the  final 
and  future  versions  of  a  product. 
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now  REVIEW  lOUBNAI 


In  order  to  reduce  and  manage  risk,  program  managers  need  to  measure  or  monitor 
contractor,  project,  and  system  performance  throughout  the  project’s  lifecycle.  This 
will  help  ensure  that  standards,  processes,  and  quality  requirements  are  being  met. 
SLAs  mandate  monitoring  of  the  quality  requirements  associated  with  product, 
process,  project,  and  post-production  quality.  If  quality  levels  are  not  met,  program 
managers  and  the  contractor  are  informed  of  the  violation  and  potential  risks;  that 
knowledge  may  lead  to  closer  monitoring  or  corrective  action  to  reduce  risk  and 
improve  quality. 

The  development  of  SLAs  provides  information  that  will  assist  the  program  manager 
in  managing  the  project’s  finances.  SLAs  help  financial  management  in  determining 
the  scope  of  the  project  (e.g.,  determining  areas  of  responsibilities  in  an  end-to-end 
performance  SLA),  identifying  business-critical  processes  and  functions,  they  help 
to  allocate  costs  among  business  units,  they  provide  justification  for  service-related 
expenditures,  and  they  help  coordinate  the  IT  strategy  with  business  strategies  (e.g., 
applying  funds  toward  services  to  support  critical  processes). 


Cotttratt  oversight  is  easier  when  SLAs  are 
estahiished  hetause  they  tan  be  used  to 
define  the  levels  of  servite  expetted, 
explain  how  the  measurement  will  be 
tondutledf  identify  responsibilities, 
and  set  forth  estalation  protedures. 


Quality  control  consists  of  the  actions  necessary  to  certify  that  desired  standards, 
processes,  and  quality  requirements  are  adhered  to  during  design,  implementation, 
and  production  (Tricker  &  Sherring-Lucas,  2001).  SLAs  are  a  quality-control 
mechanism.  SLAs  help  the  program  manager  by  contractually  mandating  that  cer¬ 
tain  quality-control  methods  be  implemented. 

Contract  oversight  is  easier  when  SLAs  are  established  because  they  can  be  used 
to  define  the  levels  of  service  expected,  explain  how  the  measurement  will  be  con¬ 
ducted,  identify  responsibilities,  and  set  forth  escalation  procedures.  The  monitoring 
mandated  by  SLAs  helps  the  program  manager  determine  whether  the  contractor  is 
meeting  the  quality  requirements.  When  determining  what  quality  metrics  are  in¬ 
cluded  in  the  SLAs,  it  is  helpful  to  determine  the  behavior  you  want  from  the  con¬ 
tractor,  and  determine  what  measurements  will  most  likely  encourage  that  behavior 
(Kendrick,  2003). 

SLAs  help  institutionalize  a  change  review  board  to  continually  review  the  SLAs, 
evaluate  new  requirements,  and  ensure  maintenance  actions  do  not  affect  the  SLAs. 
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The  change  review  board  ensures  that  only  authorized  changes  are  enacted,  that 
changes  are  tested  against  quality  levels,  that  changes  conform  to  architectural 
constraints,  and  those  changes  are  properly  documented.  SLAs  can  also  be  written 
to  specify  quality  requirements  that  deal  specifically  with  the  accuracy  or  effective¬ 
ness  of  configuration  identification,  configuration  control,  configuration  account¬ 
ing,  and  configuration  audits. 

In  addition,  SLAs  provide  a  basis  for  common  understanding  of  the  services  that 
will  be  performed,  the  levels  of  service  to  be  expected,  how  they  will  be  measured, 
as  well  as  define  the  responsibilities  of  both  parties  to  an  SLA.  Both  parties  must 
mutually  agree  upon  contractual  SLAs,  or  there  will  never  be  a  contract.  It  is  com¬ 
monplace  to  negotiate  on  the  services  and  the  performance  levels  that  are  requested 
and  ultimate^  agreed  upon.  An  SLA  should  contain  a  defmition  of  service  require¬ 
ment  that  is  both  achievable  by  the  provider  and  affordable  by  the  customer.  The 
customer  and  the  ESP  must  also  define  a  mutually  acceptable  set  of  indicators  of 
the  quality  of  service  (Sturm  et  al.,  2000).  Note  that  SLAs  can  and  should  be  modified 
throughout  the  lifecycle  of  a  system  as  requirements  change,  technology  improves, 
and  efficiencies  are  gained. 


The  use  of  SLAs  helps  to  ensure  that  tustomer 
expettations  are  heing  met  by  establishing 
performante  and  quality  oblertives,  and  providing 
the  metrks  and  a  measurement  methodoiogy  to 
ensure  that  those  requirements  are  being  met. 


The  use  of  SLAs  helps  to  ensure  that  customer  expectations  are  being  met  by 
establishing  performance  and  quality  objectives,  and  providing  the  metrics  and  a 
measurement  methodology  to  ensure  that  those  requirements  are  being  met.  An 
important  part  of  customer  service  is  monitoring  the  performance  of  the  system  to 
ensure  that  it  is  supporting  the  critical  business  processes  in  a  manner  acceptable  to 
the  customer.  The  fact  that  SLAs  specify  what  is  acceptable  makes  it  easier  for  program 
managers  to  manage  expectations. 

SLAs  can  be  valuable  to  organizations  that  lack  the  proper  IT  skills  to  contract  with 
ESPs.  Template  SLAs,  or  pre-existing  SLAs,  which  represent  best  business  practices, 
can  be  used  as  a  starting  point  in  formulating  SLAs  that  are  unique  to  specific  appli¬ 
cations.  SLA  templates  are  a  starting  point  for  program  managers  to  develop  their  own 
SLAs.  The  templates  are  helpful  in  generating  questions  regarding  services  and  service 
levels  that  should  be  provided  to  support  or  develop  applications.  The  program  man¬ 
agers  onfy  have  to  modify  the  existing  SLAs  to  best  support  their  application. 
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ACQUlSITIOtl  REVirW  lOlIBNAI 


DEPLOYMENT  SUPPORT 

Quality  control  does  not  stop  once  a  software  product  has  been  deployed.  Quality 
requirements  still  need  to  be  applied  to  the  application  performance,  maintenance 
efforts,  and  hosting  services  throughout  its  lifecycle.  Monitoring  the  performance 
of  the  application  once  it  is  deployed  is  essential  in  quality  control  and  maintaining 
customer  satisfaction.  Much  of  the  application  performance  monitoring  in  the 
initial  phases  of  deployment  is  to  validate  product-quality  requirements  identified  in 
the  initial  requirements.  However,  in  the  deployment  environment  there  is  also  an 
emphasis  on  monitoring  system  performance  in  terms  of  resource  utilization, 
application  security,  application  reliability,  application  maintainability,  database 
performance,  and  server  performance. 

Quality  requirements  associated  with  deployed  software  take  a  holistic  view  of  the 
entire  system,  including  distributed  components.  Quality  requirements  should  be 
applied  to  network  performance,  host  environment  security,  disaster  recoveiy,  stor¬ 
age  solutions,  problem  management,  concurrent  user  management,  as  well  as  end- 
to-end  performance.  Monitoring  of  the  network,  hardware,  operating  system,  and 
the  application  not  only  assists  in  problem  resolution,  but  trend  analysis  can  indicate 
potential  problems  before  they  become  unwieldy. 

SLAs  can  be  utilized  for  maintenance  actions  when  patches  or  new  versions  need 
to  be  developed  and  deployed.  SLAs  utilized  in  the  development  phase  may  be  used 
in  some  circumstances  with  maintenance  actions. 


CONCLUSION 

SLAs  are  performance-based  instruments  for  acquiring  software.  SLAs  can  be 
used  to  specify  software  quality,  but  they  are  also  useful  for  specifying  perfor¬ 
mance  requirements.  We  have  described  how  SLAs  can  drive  product,  process, 
project,  and  deployment  quality  solutions.  SLAs  can  help  ensure  that  quality  re¬ 
quirements  are  identified  and  established  early  in  the  development  cycle  in  order 
to  be  incorporated  into  preliminary  designs.  SLAs  can  help  program  managers 
establish  quality  controls  to  monitor  and  manage  the  various  aspects  of  the  projects. 
SLAs  also  carry  sufficient  weight  through  incentives  to  focus  management  and 
contractor  attention  on  the  quality  issues  for  a  software-intensive  system  that  affect 
the  ability  of  the  acquiring  organization  to  perform  its  mission. 

Future  research  that  can  build  upon  the  foundations  set  forth  in  this  paper  in¬ 
clude  evaluating  SLAs  in  actual  contracting  to  analyze  quality  improvements, 
determining  which  quality  models  and  quality  attributes  are  best  suited  for  differ¬ 
ent  types  of  IT  systems,  and  evaluating  or  generating  tools  necessary  for  measur¬ 
ing  end-to-end  SLA  measurements,  such  as  response  time  and  availability. 

All  organizations  want  world-class  quality  levels,  but  achieving  those  quality 
levels  requires  a  holistic  view  of  quality  that  incorporates  leadership  support,  re¬ 
peatable  and  measurable  quality  processes  and  controls,  resource  planning,  vision, 
customer  support,  and  service-level  management.  Organizations  must  do  more 
than  identify  and  incorporate  quality  attributes  in  their  requirements:  they  must 
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also  monitor  quality  metrics  to  ensure  those  quality  requirements  are  being  met. 
SLAs  are  instruments  that  can  be  used  to  establish  quality  controls.  Quality  is  not 
something  that  is  inherent  in  the  development  process;  it  must  be  planned,  moni¬ 
tored  and  incorporated  as  part  of  standard  business  practices. 


CDR  Lfionard  T.  Gaines,  USN  is  a  Supply  Corps  Officer  currently 
assigned  to  the  Joint  Logistics  Operation  Center  at  the  Defense  Logistics 
Agency.  He  has  a  nnaster's  degrees  from  the  Naval  Fbstgraduate  School 
and  the  Industrial  College  of  the  Armed  Forces.  He  has  served  on 
three  ships  and  has  had  shore  assignments  in  logistics,  acquisition, 
and  information  technology.  He  is  a  member  of  the  Acquisition 
Professional  Community. 

(E-mail  address;  gaines@dla.mil) 


Dr.  James  Bret  Michael  is  an  Associate  Professor  of  Computer 
Science  and  Electrical  &  Computer  Engineering  at  the  Naval 
pDStgraduate  School.  His  research  and  teaching  interests  center  on 
distributed  computing  and  sofNvare  engineering  as  they  apply  to 
ballistic  missile  defense  and  information  warfare.  He  serves  on  several 
government,  editorial,  and  industry  advisory  boards.  Michael  has  a 
doctorate  degree  from  George  Mason  University  and  is  a  senior 
member  of  the  Institute  of  Electical  and  Electronic  Engineers  (IEEE). 

(E-mail  address:  bmichael@nps.edu) 


REFERENCES 

Aries,  I,  Banerjee,  S.,  Brittan,  M.,  Dillon,  E.,  Kowalik,  J.,  &  Lixvar,  J.  (2002, 
June).  Capacity  and  performance  analysis  of  distributed  enterprise  systems. 
Communications  of  the  Association  of  Computing  Machinery  (ACM). 

Aweya,  J.,  Ouellette,  M.,  Montuno,  D,  Dor^,  B.,  &  Felske,  K.  (2002,  January/ 
February).  An  adaptive  load  balancing  scheme  for  web  servers.  International 
Journal  of  Network  Management,  12,  3-39. 


Bass,  L.,  Clements,  R,  &  Kazman,  R.  (1998).  Software  architecture  in  practice. 
Reading,  WA:  Addison- Wesley. 

Bennett,  K.,  &  Rajlich,  V  (2000,  June).  Software  maintenance  and  evolution:  A 
roadmap.  Proceedings  of  the  Conference  on  the  Future  of  Software  Engi¬ 
neering,  Limerick,  Ireland,  73-87. 

Bertoa,  M.,  &  Vallecillo,  A.  (2002,  November).  Quality  attributes  for  COTS 
components.  Retrieved  October  12,  2004,  from  http://alarcos.inf-cr.uclm.es/ 
qaoose2002/docs  /QAOOSE-Ber-Val.pdf 

Boehm,  B.  (1991,  January).  Software  risk  management:  Principles  and  practice. 
IEEE  Software,  91,  32-41. 

Briand,  L.,  Melo,  W,  Seaman,  C.,  &  Basili,  V.  (1995,  April).  Characterizing  and 
assessing  a  large-scale  maintenance  organization.  Proceedings  of  the  Seven¬ 
teenth  International  Conference  on  Software  Engineering,  Seattle,  WA  133- 
143. 

Bubenko,  J.  (1995,  March).  Challenges  in  requirements  engineering.  Second  IEEE 
Symposium  on  Requirements  Engineering,  York,  England,  160-162. 

Coppick,  C.,  &  Cheatham,  T.  (1992,  March).  Software  metrics  for  object-oriented 
systems.  Proceedings  of  the  1992  ACM  Annual  Conference  on  Communica¬ 
tions,  Kansas  City,  MI,  317-322. 

Cross,  S.  (2002,  December).  A  quality  doctrine  for  software:  Do  it  right  the  first 
time.  Proceedings  of  the  ninth  Asia-Pacific  Software  Engineering  Conference, 
Gold  Coast,  Australia,  187-194. 

Daskalantonakis,  M.  (1992,  November).  A  practical  view  of  software  measurement 
and  implementation  experiences  within  Motorola.  IEEE  Transactions  on  Software 
Engineering,  18,  998-1010. 


300 


Department  of  Defense,  Office  of  the  Inspector  General.  (2000,  July  13).  Sum¬ 
mary  of  audits  of  acquisition  of  information  technology,  (DoD  OIG  Report 
D-2000-162).  Washington,  DC:  U.S.  Government  Printing  Office. 

Dobrica,  L.,  &  Niemela,  E.  (2002,  July).  A  Survey  on  sofbvare  architecture  analy¬ 
sis  measures.  IEEE  Transactions  on  Software  Engineering,  28{1),  638-653. 

Eager,  D.,  Vernon,  M.,  &  Zahoijan,  J.  (2001,  September/October).  Minimizing  band¬ 
width  requirements  for  on-demand  data  delivery.  IEEE  Transactions  on  Knowledge 
and  Data  Engineering,  75(5),  742-757. 

Fenton,  N.,  Krause,  R,  &  Neil,  M.  (2002,  July/August).  Software  measurement: 
Uncertainty  and  causal  modeling.  IEEE  Software,  116-122. 

Fenton,  N.,  &  Neil,  M.  (1999,  September/October).  A  critique  of  software  defect 
prediction  models.  IEEE  Transactions  on  Software  Engineering,  25(5),  675-689. 

Gama,  G.,  Meira,  W,  Carvalho,  M.,  Guedes,  D,  &  Almeida,  V.  (2001,  November). 
Resource  placement  in  distributed  e-commerce  servers.  Proceedings  of  Globecom 
2001,  San  Antonio,  TX,  1677-1682. 

Garlan,  D.  (2000,  June).  Software  architecture:  A  roadmap.  Proceedings  of  the 
Conference  on  the  Future  of  Software  Engineering,  Limerick,  Ireland,  93-101. 

Gilb,  T.  (1993).  Risk  management:  A  practical  toolkit  for  identifying  analyzing 
and  coping  with  project  risk.  Retrieved  October  12,  2004,  from  http:// 
www.result-planning.com/Download/Risk.pdf 

Gnatz,  M.,  Marschall,  E,  Popp,  G.,  Rausch,  A.,  &  Schwerin,  W.  (2003,  June).  The 
living  software  development  process.  Software  Quality  Professional,  4-16. 

Harter,  D,  &  Slaughter,  S.,  (2000).  Process  maturity  and  software  quality:  A  field 
study.  Proceedings  of  the  Twenty-first  International  Conference  on  Information 
Systems,  Brisbane,  Australia,  407-411. 

Hilbum,  T.,  &  Townhidnejad,  M.  (2000,  March).  Software  quality:  A  curriculum 
postscript.  Proceedings  of  the  Thirty-First  SIGCSE  Technical  Symposium  on  Com¬ 
puter  Science  Education,  Austin,  TX,  167-171. 

Horch,  J.  (1996).  Practical  guide  to  software  quality  management.  Boston,  MA:  Artech 
House  Publishers. 

Humphrey,  W.  (1996,  May).  Using  a  defined  and  measured  personal  software  process. 
IEEE  Software,  77-88. 


301 


Kajko-Mattsson,  M.  (1998,  April).  A  conceptual  model  of  software  maintenance. 
Proceedings  of  the  Twentieth  International  Conference  on  Software  Engineer¬ 
ing,  Kyoto,  Japan,  422-425. 

Keil,  M.,  Cule,  R,  Lyytinen,  K.,  &  Schmidt,  R.  (1998,  November).  A  framework 
for  identifying  software  project  risk.  Communications  of  the  ACM,  76-83. 

Kendrick,  T.  (2003).  Identifying  and  managing  project  risk:  Essential  tools  for  failure¬ 
proofing  your  project.  New  York:  AMACOM. 

King,  W.  (2001,  February).  Guest  editorial  developing  a  sourcing  strategy  for  IS:  A 
behavioral  decision  process  and  framework.  IEEE  Transactions  on  Engineering 
Manc^ement,  45(1),  15-24. 

Mann,  C.  (2002,  July/August).  Wty  software  is  so  bad.  MIT  Technology  Review,  33-38. 

Nogueira  de  Leon,  J.  (2000,  September).  A  formal  model  for  risk  assessment  in 
software  projects.  Unpublished  doctoral  dissertation.  Naval  Pbstgraduate  School, 
Monterey,  CA. 

McGuire,  E.  (1996).  Factors  affecting  the  quality  of  software  project  management: 
An  empirical  study  based  on  the  capability  maturity  model.  Software  Quality 
Journal,  5,  305-317. 

Moeller,  M.,  Hochstetler,  S.,  Glasmacher,  R,  Jewan,  R,  Rathre,  M.,  Polacheck,  X, 
Sampath,  V,  &  Ziller,  H.  (2000).  Tivoli  Netview  6.01  and  friends.  Austin,  TX: 
IBM  Redbook. 

Ogasawara,  H.,  Yamada,  A.,  &  Kojo,  M.  (1996,  March).  Experiences  of  software 
quality  management  using  metrics  through  the  life-cycle.  Proceedings  of  the 
Eighteenth  International  Conference  on  Software  Engineering,  Berlin,  Germany, 
179-188. 

Osmundson,  X,  Michael,  X,  Machniak,  M.,  &  Grossman,  M.  (2003,  September).  Quality 
management  metrics  for  software  development.  Information  and  Management, 
40(8),  799-812. 

Radayachee,  K.  (2002,  September).  An  interpretive  study  of  software  risk  manage¬ 
ment  perspectives.  Proceedings  of  the  Annual  Research  Conference  of  the  South 
African  Institute  of  Computer  Science  and  Information  Technologists  on  Enablement 
through  Technology,  Port  Elizabeth,  South  Africa,  118-127. 

Pressman,  R.  (2001).  Software  engineering  a  practitioner’s  approach  (5th  ed.).  New 
York:  McGraw-Hill. 


302 


Pritchett,  W.  (2001,  September).  An  object-oriented  metrics  suite  for  Ada  95. 
Proceedings  of  the  Annual  ACM  SIGAda  International  Conference  on  Ada, 
Bloomington,  MN,  117-126. 

Royce,  W.  (1990,  November).  Pragmatic  quality  metrics  for  evolutionary  soft¬ 
ware  development.  Proceedings  of  the  Conference  on  TRI-ADA  ’90,  Balti¬ 
more,  MD,  551-565. 

Schmidt,  M.  (2000).  Implementing  the  IEEE  software  engineering  standards. 
Indianapolis,  IN:  Sams. 

Shaw,  M.,  &  Garlan,  D.  (1996).  Software  architecture  perspectives  on  an  emerging 
discipline.  Upper  Saddle  River,  NJ:  Prentice  Hall. 

Standish  Group.  (2003).  Chaos.  In  The  Standish  Group  report.  West  Yarmouth,  MA: 
The  Standish  Group. 

Sturm,  R.,  Morris,  W,  &  Jander,  M.  (2000).  Foundations  of  service  level  management. 
Indianapolis,  IN:  Sams. 

Sumner,  M.  (2000,  April).  Risk  factors  in  enterprisewide  information  management 
system  projects.  Proceedings  of  the  SIGCPR  Conference  on  Computer  Personnel 
Research,  Chicago,  180-187. 

Tricker,  R.,  &  Sherring-Lucas,  B.  (2001).  ISO  9001:2000  in  brief  Oxford,  MA: 
Butterworth  Heinemann, 

U.S.  General  Accounting  Office.  (2001).  Mq/or  management  challenges  and  program 
risks:  Department  of  Defense.  (U.S.  GAO  Report  GAO  01-244).  Washington,  DC: 
U.S.  Government  Printing  Office. 

Wiegers,  K.  (2002).  Software  requirements  (2nd  ed.).  Redmond,  WA:  Microsoft  Press. 

Zuse,  H.  (1991).  Software  complexity — measures  and  methods.  Berlin,  Germany:  Walter 
De  Gruyter  &  Co. 


303 


304 


PUBLIC  AND  PRIVATE 
PARTNERSHIPS  IN  SUPPORT 
OF  PERFORAAANCE-BASED 
LOGISTICS  INITIATIVES— 
LESSONS  LEARNED 
FROM  DEFENSE  LOGISTIC 
AGENCY  PARTNERSHIPS 

DR.  GLENN  L  STARKS 


Per  Product  Support  for  the  21st  Century:  A  Program  Manager's  Guide 
to  Buying  Performance,  November,  2001  published  by  the  Defense 
Acquisition  University;  Performance- Based  Logistics  (PBL)  is  the  preferred 
approach  for  implementing  weapon  system  support.  While  PBL  initia¬ 
tives  enhance  weapon  system  support  by  employing  best  commercial 
practices  in  providing  an  integrated  and  performance-based  supply 
chain,  they  often  do  not  combine  the  best  of  government  support 
with  commercial  support.  This  results  in  dual  infrastructures,  increased 
costs,  and  other  disadvantages.  This  article  addresses  the  advantages 
of  combining  public  and  private  support  by  discussing  lessons  learned 
from  two  PBLs  where  the  Defense  Logistics  Agency  has  become  the 
source  of  supply  to  commercial  vendors  awarded  PBL  contracts  by  the 
Military  Services. 


Performance-Based  Logistics  (PBL)  contracts  enhance  the  support  of  Military 
Service  >veapon  systems  by  employing  the  purchase  of  total  or  partial  system 
support  as  an  integrated  performance  package  from  a  single  source.  Per  the 
publication  Product  Support  for  the  21st  Century:  A  Program  Manager's  Guide  to 


305 


Buying  Performance,  November,  2001  published  by  the  Defense  Acquisition 
University,  “Performance-Based  Logistics  ^BL)  is  DoD’s  [Department  of  Defense] 
preferred  approach  for  implementing  product  support.  The  PBL  is  a  strata  for 
weapon  system  life  cycle  support  that  brings  higher  levels  of  system  readiness  through 
efficient  management  and  direct  accountability”  (p.  1-4).  The  majority  of  PBL  ini¬ 
tiatives  are  contracts  to  a  single  private  company.  The  overall  goal  of  PBLs  is  to 
optimize  system  readiness.  The  contractor  is  required  to  meet  support  goals  for 
a  weapon  system  by  establishing  a  support  structure  based  on  performance  metrics 
with  clear  lines  of  authority  and  responsibility.  The  contractor  performs  logistics 
functions  that  have  been  historically  performed  by  government  personnel  while 
implementing  best  commercial  principles  and  practices. 

The  inherent  disadvantage  of  PBLs,  as  often  implemented,  is  that  they  do  not 
combine  the  support  benefits  already  in  place  within  DoD.  PBL  contractors  are  required 
to  fully  and  independently  develop  a  supply  chain  management  network  to  support 
weapon  systems  without  retying  on  current  support  systems  already  in  place  within 
the  government.  This  often  leads  to  the  creation  of  dual  support  infrastructures, 
unnecessary  costs  being  assumed  ly  the  eontractor,  and  a  negative  impact  on  small 
businesses.  These  and  other  disadvantages  are  overcome  by  the  development  of  public 
and  private  partnerships  in  the  support  of  PBL  contracts. 


Therefore,  in  addition  to  maintaining  a  level  of 
performanre  and  implementing  <ommerdal  supply 
<hain  management  best  prindples  and  prattites, 
the  PBL  iontrattor  is  also  required  to  perform 
materiel  management,  distribution,  tethnitai  data 
management,  rataloging,  and  rontrarting  fumtions 
in  obtaining  and  providing  parts  support. 


This  article  examines  two  PBL  partnerships  that  have  been  developed  between  the 
Defense  Logistics  Agency  (DLA)  and  two  PBL  contractors;  whereby,  DLA  has  become 
a  souree  of  supply  for  consumable  parts  in  support  of  the  weapon  systems  under  each 
PBL  contract.  The  lessons  learned  from  these  partnerships  are  presented  to  illustrate 
the  resulting  positive  impact  in  improving  weapon  system  readiness  and  in  reducing 
overall  costs.  These  reduced  costs  are  ultimatety  passed  on  to  the  Military  Service 
activity  awarding  the  PBL  contract. 
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TWO  PBL  EXAMPLES 

While  PBL  contracts  require  contractors  to  perform  an  array  of  services  to  improve 
weapon  system  readiness,  these  services  are  often  centered  around  providing  materiel 
within  specific  timeframes  to  ensure  parts  are  readily  available  to  ensure  weapon 
system  performance.  This  materiel  includes  both  reparable  parts  (historically  managed 
by  Military  Service  Inventory  Control  Points)  and  consumable  parts  (historically 
managed  by  DLA).  Therefore,  in  addition  to  maintaining  a  level  of  performance 
and  implementing  commercial  supply  chain  management  best  principles  and  prac¬ 
tices,  the  PBL  contractor  is  also  required  to  perform  materiel  management,  dis¬ 
tribution,  technical  data  management,  cataloging,  and  contracting  functions  in 
obtaining  and  providing  parts  support.  These  later  logistic  functions  are  inherent 
in  the  mission  of  DLA.  In  the  providing  of  consumable  parts  support,  the  con¬ 
tractor  is  required  to  both  duplicate  and  improve  support  historical^  provided  by 
DLA.  Thus  results  the  aforementioned  disadvantages  of  PBLs  by  creating  dual 
systems  of  management  by  ignoring  the  capabilities  already  available  in  the  public 
sector.  The  two  PBLs  analyzed  in  this  study  illustrate  how  these  disadvantages 
can  be  overcome. 


From  the  Navy's  standpoint,  the  iontrattor  has 
the  ultimate  responsihility  in  providing  total 
supply  thain  management  support 


The  Navy  awarded  a  PBL  contract  to  Lockheed  Martin  in  support  of  the  S-3 
aircraft.  Lockheed  Martin  subcontracted  materiel  support  to  Logistics  Services  Inter¬ 
national  (LSI)  as  a  third  party  logistics  (3PL)  parts  provider  The  Navy  also  awarded 
a  repair  contract  to  Pratt  and  Whitney  in  support  of  the  J52  engine.  Although  repair 
contracts  do  not  have  requirements  as  extensive  as  PBLs,  they  share  many  of  the 
similarities  and  require  contractors  to  perform  many  of  the  same  functions  as  PBL 
contractors.  A  full  PBL  contract  is  expected  to  be  awarded  for  the  J52  once  the  repair 
contract  is  completed,  to  include  extended  services  and  materiel  support.  In  both 
examples,  the  Navy  has  transitioned  its  reliance  on  materiel  support  from  historically 
governmental  sources  of  supply  (DLA  and  Navy  oiganizations)  to  contractor  sup¬ 
port.  Also  in  both  cases,  DLA  has  established  partnerships  with  each  PBL  contractor 
to  provide  consumable  parts.  Under  the  S-3  PBL,  DLA  provides  support  for  1,087 
items.  The  DLA  conducted  an  initial  supply  screen  of  the  items,  and  made  advance 
buys  vdiere  needed.  The  PBL  vendor  was  advised  of  each  item’s  price  and  stock 
position  in  relation  to  the  forecast.  Under  the  J52  PBL,  DLA  provides  support  for 
161  items.  The  DLA  competed  to  become  a  qualified  source  for  the  items  based  on 
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quality,  delivery,  and  performance  standards.  In  both  cases,  parts  were  prepositioned 
based  on  where  the  actual  work  was  being  performed  by  the  PBL  vendor. 

Although  each  of  these  PBL  contractors  obtains  consumable  parts  support 
from  DLA,  they  are  in  no  way  relieved  from  the  contractual  performance  re¬ 
quirements  in  their  respective  contracts  with  the  Navy.  From  the  Navy’s  stand¬ 
point,  the  contractor  has  the  ultimate  responsibility  in  providing  total  supply  chain 
management  support.  They  are  also  still  held  accountable  for  establishing  the 
most  effective  and  efficient  cost  structure  to  meet  the  needs  of  the  government. 
The  DLA  must  then  compete  as  a  source  of  supply  and  proves  its  ability  to  pro¬ 
vide  support  to  the  PBL  contractor  that  will  fit  within  this  efficient  cost  stnicture. 
Accomplishing  this  is  a  challenging  task  that  has  resulted  in  the  Agency  itself 
becoming  more  effective  and  efficient  in  performing  its  historical  duties  as  a  manager 
of  consumable  parts  and  providing  services  connected  to  providing  materiel.  The 
following  eight  lessons  learned  from  both  of  these  examples  illustrate  the  advan¬ 
tages  of  DLA  and  PBL  contractors  establishing  partnerships  to  support  PBL  ini¬ 
tiatives.  These  partnerships  are  instituted  via  of  Memorandums  of  Understanding 
(MOUs)  or  Performance-based  Agreements  (PBAs)  that  outline  the  terms  and 
conditions  that  each  party  will  adhere  to  (a  sample  is  provided  in  Appendix  A). 


LESSONS  LEARNED 

1.  Creates  Partnering  versus  Dual  Infrastructures— With  PBLs  where  the  vendor 
does  not  partner  with  DLA  to  obtain  consumable  support,  the  contractor  must 
establish  partnerships  with  other  contractors  to  obtain  parts,  personnel  to  manage 
the  acquisition  process,  and  even  a  distribution  network  to  store  and  move  materiel. 
Under  the  S-3  and  J52  PBLs,  the  vendors  simple  identify  their  requirements  to 
DLA.  DLA’s  personnel  fill  the  requisitions,  establish  long-term  agreements  with 
commercial  vendors,  and  DLA  already  has  a  worldwide  storage  and  distribution 
network  in  place.  To  ensure  that  the  contractor  is  able  to  meet  the  requirements 
outlined  in  the  PBL  contract,  the  contractor  and  DLA  partner  on  sources  of  supply 
and  forecasting,  and  DLA  sets  performance  standards  compatible  with  those  in 
the  PBL  contract.  While  these  performance  standards  vary  per  each  PBL  agree¬ 
ment,  some  examples  include  ensuring  parts  availability  between  90  percent  and 
100  percent,  ensuring  100  percent  adherence  to  quality  standards,  and  processing 
and  shipping  materiel  within  24  hours. 

2.  Reduces  Materiel  Costs— Many  of  the  items  used  on  the  S-3  and  J52  are  also 
used  on  other  weapon  systems,  both  within  the  Navy  and  on  systems  belonging 
to  the  Air  Force  and  Army.  Because  DLA  buys  these  parts  in  volume  to  support 
its  customers  across  the  Department  of  Defense,  it  can  obtain  much  better 
prices  than  a  single  PBL  vendor  purchasing  small  quantities  of  materiel  in 
support  of  a  single  PBL  initiative.  DLA  is  also  able  to  obtain  parts  cheaper  than 
PBL  contractors  because  it  has  established  long-term  agreements  in  place  with 
parts  providers. 
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These  agreements  leverage  the  buying  pcfwer  of  DLA  across  the  entire  population 
of  National  Stock  Numbers  (NSNs)  on  these  long-term  contracts,  leading  to 
cheaper  prices  for  each  individual  item.  Since  these  agreements  are  put  in  place 
for  up  to  ten  years,  lower  prices  are  maintained  over  long  periods  of  time.  For 
example,  over  the  past  year  the  total  price  for  the  items  used  on  the  J52  PBL 
has  decreased  4.78  percent.  Lastly,  DLA’s  price  encompasses  the  entire  cost 
of  maintaining  a  complete  supply  chain,  from  initial  cataloging  through  distri¬ 
bution.  Unlike  customers  that  rely  on  a  single  contractor  for  total  support,  DLA’s 
customer  are  charged  prices  for  materiel  that  are  not  affected  by  shifts  in  per¬ 
sonnel  or  other  changes  in  inftastmcture  due  to  varying  customer  demands  because 
of  DLA’s  ability  to  absorb  these  variations. 

3.  Holds  DLA  More  Accountable — ^In  supporting  the  S-3  and  J52,  DLA  can  be 
held  to  concrete  parts  support  and  performance  requirements  because  the  PBL 
vendor  is  required  under  the  PBL  contract  to  definitively  define  their  require¬ 
ments.  In  order  to  maximize  efficiency  and  minimize  costs,  PBL  contractors 
must  only  forecast  and  order  what  they  actual^  need.  Any  overages  will  result 
in  excess  costs  that  cannot  be  recouped,  and  any  forecasts  below  what  is  needed 
will  result  in  the  contractor  not  being  able  to  meet  the  terms  of  the  contract,  thus 
being  subject  to  monetary  penalties  from  the  Military  activity  that  awarded  the 
PBL  contract.  With  more  accurate  forecasts,  DLA  is  able  to  procure  materiel  in 
advance  of  the  contractor’s  need  and  even  preposition  materiel  at  or  near  the 
point  of  use.  The  PBL  contractor  minimizes  costs  by  not  having  to  assume  any 
costs  until  they  actually  requisition  the  materiel  for  point  of  use  consumption. 

4.  Enhances  Commercial  Partnerships— The  DLA  has  developed  Strategic  Supplier 
Alliances  with  both  Lockheed  Martin  and  Pratt  and  Whitney  (copies  of  the  DoD 
Strategic  Supplier  Alliance  Project  Guidebook  and  DLA  Strategic  Supplier  Alliance 
(SSA)  charters  can  be  found  at  http://www.dla.mil/j-3/j-336/logisticspolicy/j-3312/ 
webpage%20ssa.htm).  These  alliances  are  strat^ic  partnerships  whereby  NSNs 


TABLE  1.  DEFENSE  LOGISTICS  AGENCY  (DLA) 
STRATEGIC  SUPPLIER  ALLIANCE  (SSA)  PARTNERS 


DLA  Aviation  Strategic  Supplier  Alliance  (SSA)  Partners 

Lockheed  Martin 

Moog  Inc. 

Pratt  and  Whitney 

Aircraft  Braking  Sysytems 

Boeing 

Canadian  Commercial 

Hamilton  Sundstrand 

Eaton  Corporation 

Parker  Hannifin 

Rolls  Royce 

Goodrich  Corporation 

Sikorsky 

Textron/Bell  Helicopter 

BAE  Systems 

Northrop  Grumman 

General  Electric 

1  Honeywell  I 
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managed  by  DLA  that  are  solely  sourced  from  these  vendors  are  placed  on 
long-term  corporate  contracts  with  performance  metrics.  The  Navy,  Army, 
and  Air  Force  may  also  join  these  partnerships  and  add  their  managed  NSNs 
to  the  alliance  as  well. 

For  example,  the  Air  Force  has  partnered  with  DLA  in  the  SSA  with  Pratt  and 
Whitney.  By  including  PBL  partnerships  under  the  umbrella  of  the  alliance, 
the  partnership  between  the  government  and  private  enterprise  is  further  strength¬ 
ened  ty  broadening  the  government’s  buying  power  with  each  alliance  partner. 
Including  Lockheed  and  Pratt  and  Whitney,  DLA  has  established  17  SSAs 
with  its  top  20  Aviation  suppliers  (see  Table  1  for  complete  list).  Almost  30,000 
items  are  under  long-term  contracts  with  these  suppliers.  These  are  the  same 
vendors  that  are  being  awarded  the  majority  of  PBL  contracts  for  aviation 
systems  by  the  Military  Services.  These  partnerships  further  enable  DLA  to 
leverage  its  buying  power  in  support  in  PBL  initiatives  across  all  of  these  con¬ 
tractors  and  their  individual  divisions. 


Of  the  1,248  NSNs  €oHe€tively  being  supported 
under  the  5-3  and  J52  PBls,  utmost  800  are 
provided  by  DUTs  SSA  partners. 


Other  alliances  have  been  established  for  DLA’s  top  suppliers  of  parts  for  its 
Land  and  Maritime  customers.  Of  the  1,248  NSNs  collectively  being  supported 
under  the  S-3  and  J52  PBLs,  almost  800  are  provided  by  DLA’s  SSA  partners. 
The  remaining  population  of  NSNs  are  predominantly  competitive  items 
(multiple  sources)  and  thus  cannot  be  added  under  SSA  arrangements,  but 
many  are  also  supported  by  other  types  of  long-term  contracting  arrangements. 

5.  Gives  Customers  One  Source  of  Supply— As  stated  earlier,  the  PBL  vendor 
is  not  relieved  from  the  performance  metrics  outlined  in  the  PBL  contract,  even 
when  thty  use  a  government  supplier  to  obtain  materiel.  Therefore,  the  Navy  still 
ultimately  obtains  integrated  support  from  one  commercial  entity,  rather  than  a 
multitude  of  government  and  commercial  sources.  All  of  the  benefits  that  are 
obtained  from  the  PBL  vendor  using  DLA  as  a  source  of  supply  are  thus  passed 
on  to  the  Navy.  If  DLA  cannot  provide  parts  within  the  performance  parameters, 
the  PBL  vendor  is  free  to  obtain  these  parts  from  commercial  sources.  This  creates 
a  financial  incentive  for  DLA  to  have  parts  available  to  meet  the  PBL  vendor’s 
forecast  within  specified  performance  parameters  in  able  to  remain  a  viable  source 
of  supply.  With  the  J52  PBL,  DLA’s  performance  in  providing  parts  has  sur¬ 
passed  the  performance  metrics  outlined  in  the  PBL  contract. 
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6.  Increases  Availability  of  Parts— With  DLA’s  SSAs,  other  long-term  agree¬ 
ments  and  leveraged  buying  power,  the  instances  of  parts  availability  is  greatly 
increased  beyond  the  availability  the  PBL  vendors  would  have  obtained  if 
they  had  sought  to  obtain  the  parts  themselves  from  the  commercial  market. 
The  DLA  procures  its  parts  from  the  somces  the  Military  Services  dictate. 
For  the  collective  population  of  DLA  NSNs  under  the  S-3  and  J52  PBL, 
there  are  over  100  individual  sources  for  these  NSNs.  If  these  PBL  contrac¬ 
tors  had  not  partnered  with  DLA,  they  would  have  had  to  develop  individual 
partnerships  with  these  vendors  rather  than  simply  obtaining  support  from 
just  DLA. 

On  other  PBL  contracts  \nhere  vendors  have  declined  to  partner  with  DLA  and 
strictly  rely  on  commercial  sources,  there  have  been  recurring  cases  of  where 
they  have  been  unable  to  obtain  parts  or  have  obtained  them  at  prices  far  greater 
than  DLA’s.  While  these  PBL  vendors  have  experienced  delays  in  obtaining  parts, 
DLA  has  had  the  materiel  readily  available.  Because  the  PBL  vendor  had  initially 
declined  to  partner  with  DLA,  the  stock  DLA  did  have  available  was  procured 
to  meet  the  demands  of  its  other  customers.  Providing  the  materiel  to  the  PBL 
vendor  then  creates  overall  support  shortages  because  of  a  lack  of  the  PBL  vendor’s 
continuous  requirements. 

7.  Allows  PBL  Vendor  To  Focus  on  Services— Since  the  vendors  allow  DLA  to 
concentrate  on  providing  consumable  parts  support  for  the  S-3  and  J52,  they  are 
then  free  to  concentrate  their  energies  on  providing  enhanced  services  in  other 
areas.  While  these  vendors  have  expertise  in  providing  best  commercial  practices 
toward  repairaWe  parts  supply  chain  management,  DLA  has  expertise  in  providing 
consumable  support  gained  from  decades  of  experience.  The  net  effect  is  utilizing 
the  best  of  the  private  and  public  sector  in  developing  a  support  system  to  enhance 
support  to  the  Navy. 

8.  Ensures  Survivability  of  Small  Businesses— Many  of  the  NSNs  on  the  S-3  and 
J52  PBL  have  been  historically  provided  by  small  businesses.  Over  the  past  two 
years,  small  businesses  have  been  awarded  $7.7  million  in  contracts  from  DLA 
from  this  population  of  items.  While  DLA  is  mandated  to  ensure  that  these  items 
are  procured  from  small  businesses  in  order  to  ensure  the  survival  of  these 
businesses,  PBL  contractors  are  not  held  to  the  same  federal  small  business 
requirements  and  goals  as  DoD  agencies.  Obtaining  consumable  parts  support 
from  DLA  ensures  that  small  businesses  do  not  lose  a  substantial  portion  of  their 
income,  as  has  been  the  case  with  other  PBLs  where  the  vendor  has  declined  to 
obtain  parts  from  DLA.  By  ensuring  these  small  businesses  remain  viable  sources, 
DLA  is  also  ensuring  there  is  an  active  industrial  base  for  future  support. 
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SUMMARY 

The  above  lessons  learned  outline  the  benefits  of  PBL  vendors  partnering  with 
DLA  in  providing  support  to  weapon  systems  under  PBL  contracts.  The  elimi¬ 
nation  of  dual  infrastructures,  reductions  in  materiel  costs,  increased  government 
accountability,  enhanced  commercial  partnerships,  a  single  source  to  the  Mili¬ 
tary  Service,  increased  parts  availability,  and  increased  focus  by  the  PBL  vendor 
and  benefits  to  small  businesses  have  all  contributed  to  the  success  of  enhanced 
support  to  the  J52,  S-3  and  other  PBLs  where  DLA  has  partnered  with  private 
companies. 

The  Military  Services  are  also  recognizing  the  benefits  of  these  DLA/private 
company  partnerships  and  are  including  DLA  personnel  on  their  Planning  and 
Intergrated  Process  Teams  as  they  expand  the  use  of  PBLs  to  enhance  weapon  system 
support.  Weapon  system  Program  Managers  and  acquisition  personnel  should  engage 
DLA  early  in  the  planning  process  to  fully  understand  what  benefits  DLA  can 
offer.  DLA  is  aHe  to  tailor  many  of  its  services  to  the  individual  needs  of  each 
PBL  support  program.  The  PBL  contractors  should  also  engage  DLA  early  in  the 
PBL  process,  and  even  to  outline  what  support  will  come  from  DLA  as  part  of 
their  responses  to  PBL  solicitations.  As  discussed  in  this  article,  int^rating  the 
best  of  the  public  and  private  sector  produces  benefits  to  all  parties  involved  and 
specifically  improves  the  ultimate  support  to  the  warfighter. 


Glenn  L  Starks,  Ph.D.,  is  Chief  of  the  Planning  and  Requirements 
Division  at  Defense  Supply  Center,  Richmond,  VA.  He  has  oversight  of 
development  and  tracking  of  strategic  initiatives  that  enhance  support 
to  the  Aviation  customers  of  the  Defense  Logistics  Agency,  including 
Rgrforrnonce -Based  bgistics  initiatives.  He  holds  a  bachelor's  degree 
in  business  administration,  a  master's  degree  in  management,  a 
master's  certificate  in  project  management  and  a  Ph.D.  in  public 
policy  and  analysis. 

(E-mail  address:  glenn.starks@dla.mil) 
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SAMPLE  MEMORANDUM  OF  UNDERSTANDING 
DEFENSE  LOGISTICS  AGENCY  AND  PBL  VENDOR 


This  Memorandum  of  Understanding  (this  “MOU”)  is  made  and  entered  into  as  of 

the  1st  day  of _ (the  “Effective  Date”),  and  between  PBL 

VENDOR  NAME  and  the  Defense  Logistics  Agency  (DLA),  either  or  both  of  which 
may  be  hereinafter  referred  to  as  the  “Party”  or  the  “Parties,”  respectively. 

I.  PURPOSE 


In  accordance  with  Contract _ between  PBL  VENDOR  NAME 

and  the  Military  Service,  PBL  VENDOR  NAME  is  authorized  to  obtain  parts  fi-om 
DLA  to  support  the  weapon  system(s)  name  The  purpose  of  this  MOU  is  to  confinn 
a  basic  understanding  of  the  Parties  regarding  the  process  of  PBL  VENDOR  NAME 
providing  a  forecast  and  ordering  the  parts  and  DLA  providing  the  parts. 

II.  TERM 

This  MOU  shall  commence  as  of  the  Effective  Date  and  terminate  onty  at  the 

convenience  of  both  Parties  or  the  expiration  of  Contract - , 

whichever  occurs  first.  Termination  intent  between  DLA  and  PBL  VENDOR  NAME 

before  the  expiration  of  Contract _ will  be  communicated  in  writing. 

Upon  termination,  both  Parties  unconditionally  waive  any  charges  against  either  Party 
because  of  termination  of  the  MOU  and  release  each  other  from  all  obligations  under 
the  MOU. 


III.  ORDERING 


Per  the  terms  of  the  Contract _ ,  all  requisitions  submitted  to 

DLA  will  be  done  via  Military  Standard  Requisitioning  and  Issue  Procedures 
(MILSTRIP)  or  via  the  DoD  Electronic  Mall  (EMALL).  Requisitions  may  be  sub¬ 
mitted  via  automated  requisitioning  processing  through  DoD  MILSTRIP  auto¬ 
mated  routing,  or  directly  to  DLA  Inventoiy  Control  Points  (ICPs)  via  telephone, 
fax,  or  mail.  DLA  agrees  to  provide  PBL  VENDOR  NAME  ai^  training  needed 
on  the  DoD  MILSTRIP  or  EMALL  requisitioning  process.  A  valid  Department 
of  Defense  Activity  Address  Code  (DoDAAC)  will  identify  all  requisitions  sub¬ 
mitted  by  PBL  VENDOR  NAME  as  identified  for  use  by  PBL  VENDOR  NAME 
from  the  Military  Service(s).  PBL  VENDOR  NAME  agrees  that  all  materiel 
provided  by  DLA  to  PBL  VENDOR  NAME  in  support  of  Contract 
_ will  only  be  used  in  the  providing  services  and  parts  sup¬ 
port  under  this  bilateral  basic  ordering  agreement.  DLA  will  provide  parts  pur¬ 
chased  from  approved  sources  per  the  Military  Service  Engineering  Support 
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Activity.  DLA  will  process  requisitions  received  in  accordance  with  MILSTRIP 
prioritization  policies  with  adherence  to  DoD’s  Uniform  Material  Movement  and 
Issue  Priority  System  (UMMIPS)  timeframes  based  upon  the  requisition  priority, 
at  a  minimum. 


IV  PROCESSING  ORDERS 

DLA  agrees  to  process  all  requisitions  received  in  the  most  expeditious  manner 
possible.  DLA  will  track  these  requisitions  from  the  date  of  submission  to  the  date 
of  shipment,  and  provide  requested  status  of  these  requisitions  within  24  hours.  If 
stock  is  not  immediate^  available  to  fill  the  requisition  from  stock  on  hand  or  via 
contract  with  another  entity,  DLA  agrees  to  expedite  delivery  of  materiel  for  delivery 
to  meet  Military  Service  requirements  for  materiel  ordered  under  Contract 

- - — .  subsequent  to  a  review  and  agreement  on  any  additional  charges 

that  may  result.  DLA  agrees,  whenever  possible  and  based  upon  cost  and  demands 
from  other  users,  to  position  stock  for  the  most  expeditious  delivery  to  the  end  user. 
This  may  involve  positioning  materiel  from  current  depot  locations  or  outlining 
shipping  instructions  in  contractual  agreement  terms  with  vendors. 

V.  TRAINING 

DLA  agrees  to  provide  PEL  VENDOR  NAME  access  and  training  to  DLA  inventory 
systems  for  the  purposes  of  requisition  submission,  status,  tractability,  and  inventory 
visibility.  All  access  and  training  provided  are  subject  to  DoD  security  requirements. 

VI.  INFORAAATION  SHARING 

In  support  of  this  PEL,  PEL  VENDOR  NAME  and  DLA  agree  to  share  information 
to  enhance  the  long-term  support  of  the  Military  Service.  PEL  VENDOR  NAME 
will  provide  a  quarterly  forecast  (March,  June,  September,  December)  to  DLA. 

VII.  CHANGES 

Both  Parties  will  review  this  agreement  at  least  annually,  and  both  Parties  can  make 
modifications  at  arry  time  upon  agreement.  Any  agreements  made  outside  of  the  terms 
stated  within  this  agreement  are  only  effective  upon  modification,  issuance,  and 
signature  of  a  revised  MOU. 

VIII.  CONFIDENTIALITY 

Any  confidentiality  obligation  will  be  established  in  a  separate  Proprietary  Information 
Agreement  (“PIA”).The  PIA  shall  survive  any  termination  or  expiration  of  this  MOU 
and  remain  in  full  force  and  effect. 
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PURI  in  AMD  PRIVATE  PARTMEBSHils  IN  SPPI^Rr.  QE  i 


IX.  RELATIONSHIP  OF  PARTIES;  NO  RIGHTS  CONFERRED 

Nothing  in  this  MOU  shall  be  construed  as  giving  rise  to  a  relationship  among 
or  between  the  Parties  of  prime  contractor  and/or  sub-contractors,  employer  and 
employee,  partners,  agency,  or  joint  venturers.  Nothing  contained  in  this  MOU 
shall  be  construed  as; 

1.  Granting  or  conferring  any  right  to  use  ary  information  or  know  how  that  a  Party 
shall  elect  to  furnish  hereunder  except  as  expressly  authorized  in  this  MOU;  or 

2.  Conferring  any  license  or  right  with  respeet  to  ary  trademark,  trade  or  brand 
name,  the  corporate  name  of  either  Party  hereto  or  the  corporate  name  of  a 
subsidiary  of  either  Party  hereto  or  of  any  other  name  or  mark  or  ary  contraetion, 
abbreviation,  or  simulation  thereof. 

X.  COUNTERPARTS 

This  MOU  may  be  executed  in  one  or  more  counterparts,  including  by  facsimile, 
each  of  which  shall  be  deemed  an  original,  but  all  of  which  together  shall  eonstitute 
one  and  the  same  instrument. 

IN  WITNESS  WHEREOF,  the  Parties  have  caused  this  agreement  to  be  duty  executed 
by  their  authorized  representatives. 


PEL  VENDOR  NAME 


Defense  Logistics  Agency 


Name: 


Name; 
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PERFORAAANCE-BASED 
SERVICE  ACQUISITION 
(PBSA),  A-76  AND 
PERSONAL  SERVICES— 
A  CAUTIONARY  NOTE 

EDWARD  ALLEN  FRIAR 


The  concurrent  emphasis  on  acquiring  services  using  Performance -Based 
Service  Acquisition  (PBSA)  and  the  new  A-76  competitive  sourcing  pro¬ 
cedures  gives  rise  to  some  potentially  conflicting  goals  that  acquisition 
personnel  need  to  be  aware  of  in  order  to  avoid  personal  service  con¬ 
tracts.  A  contract  for  services  can  become  a  personal  services  contract 
either  by  the  way  it  is  written  or  by  the  way  it  is  administered,  but  proper 
training  and  planning  can  help  avoid  this  pitfall.  Acquisition  and  con¬ 
tracting  personnel  need  to  be  informed  about  what  constitutes  personal 
services  and  aware  of  this  limitation  as  it  applies  to  managing  PBSA 
contracts.  This  article  seeks  to  further  define  personal  services  and  of¬ 
fers  some  suggestions  for  consideration  when  writing  a  performance 
work  statement  (PWS)  or  statement  of  objectives  (SOO)  for  a  PBSA. 


The  Department  of  Defense  (DoD)  now  buys  more  services  than  goods  and  this 
trend  is  expected  to  continue  into  the  foreseeable  future  (Gansler,  2001).  All 
executive  departments  are  required  to  use  Performance-based  Service  Acquisition 
(PBSA)  to  the  maximum  extent  when  acquiring  services  (Federal  Acquisition 
Regulation  [FAR],  2004,  37.102,  p.  831).  The  President  has  directed  all  executive 
departments  to  contract  out  commercial  services  using  the  Office  of  Management 
and  Budget’s  (0MB)  Revised  Circular  A-76,  May  29,  2003,  vAiich  outlines 
procedures  designed  to  subject  commercial  activities  performed  by  government 
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personnel  to  the  forces  of  competition  thereby  ensuring  taxpayers  get  maximum 
value  for  their  tax  dollars  (Office  of  Management  and  Budget  [0MB],  2003).  As 
of  2003,  more  than  850,000  government  jobs  have  been  identified  under  the 
Federal  Activities  Inventory  Reform  (FAIR)  Act  as  commercial. 

The  Federal  Acquisition  Regulation  (FAR)  at  Part  37.104  has  long  prohibited 
personal  services  contracts,  and  the  previous  Circular  A-76  also  stated  that  the  Circular 
did  not  authorize  contracts  which  establish  an  employer-employee  relationship  between 
the  government  and  contractor  employees,”  i.e.,  personal  services  (0MB,  1983,  Para. 
7(c)  (5)).  This  language  has  been  removed  from  the  revised  Circular  A-76,  but  any 
contract  awarded  as  a  result  of  this  process  would  still  be  covered  by  the  FAR,  which 
prohibits  personnel  services  contracts.  Now  with  the  increasing  emphasis  on 
competitive  sourcing  of  government  jobs  using  perfomiance-based  contracting  methods 
that  allow  only  the  statement  of  the  outcome  or  goals  to  be  achieved  under  the  contract, 
there  is  an  increased  potential  for  violating  the  prohibition  against  contracting  for 
personal  services.  Acquisition  and  contracting  personnel  need  to  be  informed  about 
what  constitutes  personal  services  and  aware  of  this  limitation  as  it  applies  to  managinc 
PBSA  contracts. 


Now  with  the  imreasing  emphasis  on  iompetitive 
sourcing  of  government  iohs  using  performance- 
based  contracting  methods  that  ailow  only  the 
statement  of  the  outcome  or  goals  to  he  achieved 
under  the  contract,  there  is  an  increased 
potential  for  violating  the  prohibition 
against  contracting  for  personal  services. 


According  to  FAR  37.104,  “personal  services  contracts  are  characterized  by  the 
emplqyer-emplqyee  relationship  they  create  between  the  government  and  the  contractor 
personnel  (FAR,  2004,  37.104(a),  p.  832).  ‘Agencies  of  the  federal  government  are  not 
allowed  to  award  personal  services  contracts  unless  specifically  authorized  by  statute  (5 
United  States  Code  [U.S.C.]  3109)  to  do  so”  (FAR,  2004,  37.104(b),  p.  833).  But 
what  actually  constitutes  an  employer-employee  relationship  and  thus  personal  services? 
The  FAR  says  an  employer-employee  relationship  is  created  when  the  service  contract’s 
terms,  or  the  manner  of  the  contract’s  administration,  subject  contractor  personnel  to 
relative^  continuous  supervision  and  control  of  a  government  officer  or  emplojee.  Merely 
ordering  a  specific  performance  or  result,  wnth  the  right  to  accept  or  r^ect  the  w^rk,  is 
not  that  type  of  siqiervision  (FAR,  2004).  Each  contract  is  to  be  judged  on  its  own  m^ts 
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with  the  primary  question  being:  Will  the  government  exercise  relatively  continuous 
supervision  and  control  over  contractor  personnel  performing  the  contract  (EAR,  2004)? 

In  FAR  37.104(d)  there  are  some  descriptive  elements  that  should  be  used  in  as¬ 
sessing  whether  or  not  a  proposed  contract  is  for  personal  services,  these  include: 

1 .  Is  performance  on  site?  (i.e.,  on  a  Government  Installation) 

2.  Are  the  principal  tools  and  equipment  furnished  by  the  government? 

3.  Are  the  services  being  performed  by  the  contractor  directly  related  to  the 
accomplishment  of  the  agencies  assigned  mission  or  function? 

4.  Are  comparable  services  being  performed  in  the  same  agency  by  civil  service 
personnel? 

5.  Will  the  need  for  this  type  service  be  expected  to  last  beyond  1  year? 

6.  Does  the  nature  of  the  service  being  provided  reasonably  require  government 
direction  or  supervision  of  contractor  employees  in  order  to  adequately  protect  the 
government’s  interest,  retain  control  of  the  function  involved,  or  retain  personal 
responsibility  for  the  function  by  a  duty  authorized  Federal  officer  or  employee? 
(FAR,  2004,  37.104(d),  p.  833) 

If  you  have  a  services  contract  with  some  of  these  elements  then  you  potentially 
have  a  personal  services  type  contract.  Yon  need  to  proceed  with  caution,  re-evaluate 
your  performance  work  statement  (PWS)  or  statement  of  objectives  (SOO)  to  ensure 
that  government  personnel  will  not  be  directing  or  supervising  contractor  employees 
in  order  to  accomplish  the  contract  performance  requirements.  In  1999,  the  General 
Accounting  Office  (GAO)  denied  a  protest  from  Encore  Management  Company  (see 
B-278903.2)  over  the  cancellation  of  a  solicitation  for  clerical  and  administrative  support 
services  where  the  government  agency’s  actual  requirement  was  for  personal  services. 
As  stated  in  the  decision,  “Although  the  incumbent  contract  started  off  small  and 
included  temporary,  short  term  positions  for  a  limited  portion  of  the  agency,  the  re¬ 
quirements  quickly  grew  into  requirements  for  permanent  clerical  and  administrative 
positions  throughout  the  agency.  The  contractor’s  personnel  in  these  positions  worked 
at  the  agency’s  offices  alongside  government  employees  performing  the  same  or  simi¬ 
lar  work  and  using  government  supplies  and  equipment.  Government  managers  super¬ 
vised  contractor  personnel  by  directing,  reviewing,  and  approving  their  work”  (GAO, 
1999,  p.  4). 

The  GAO  went  on  to  say  that  cancellation  of  this  solicitation  was  reasonable  and 
proper  because,  “the  agency  no  longer  needs  temporary  personal  services  for  short 
term  positions;  rather,  the  onfy  purpose  of  this  contract  is  to  satisfy  the  agency’s  needs 
for  full  time  permanent  staff”  (GAO,  1999,  p.  4).  The  point  here  is  that  simpfy  because 
an  agency  has  a  need  for  additional  manpower  that  is  commercial  in  nature  does  not 
relieve  the  agency  from  the  prohibition  on  contracting  for  personal  services. 
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Although,  the  new  A-76  process  does  not  specifically  prohibit  the  awarding  of 
personal  services  contracts,  the  implication  would  seem  to  be  that  personal  services 
would  not  be  permitted  unless  they  fall  under  one  of  the  exceptions  allowed  in  the 
FAR  or  in  other  law.  It  is  up  to  the  contracting  officer  and  perfomiance  work  statement 
team  to  ensure  that  the  PWS/SOO  does  not  contain  personal  service  requirements. 
Contracting  personnel  should  use  the  guidance  in  FAR  37,  as  discussed  above,  in 
determining  if  the  solicitation  contains  requirements  for  personal  services.  If  it  does, 
they  should  be  deleted  or  rewritten  to  make  them  non-personal  services. 

For  example,  in  a  recent  opinion  hy  the  Chief  Counsel  of  the  Legal  Office,  U.S. 
Army  Communications  and  Electronics  Command  (CECOM),  it  was  determined  that, 
it  appears  that  contracting  for  administrative  or  secretarial  services  traditionally  pro¬ 
vided  by  the  Legal  Office  support  staff  would  be  difficult  since  obtaining  those  ser¬ 
vices  from  contractor  personnel  without  relatively  continuous  supervision  does  not 
appear  feasible  (Szymanski,  2001,  p.  5).  Not  only  would  the  inherent  nature  of  this 
service  require  continuous  supervision^  but  this  is  an  on  going  requirement  expected 
to  last  more  than  one  year,  the  work  would  be  performed  on-site,  the  principal  tools 
and  equipment  would  be  furnished  by  the  government,  and  comparable  services  are 
performed  in  the  agency  using  civil  service  emplcyees.  A-76  is  about  competitive 
sourcing  not  about  filling  government  employee’s  chairs  with  contractors. 


A-76  is  about  tompefitive  sourting  not 
about  fating  government  employee's 
thairs  with  iontrattors. 


Although  it  may  seem  fairly  straight  forward  to  determine  if  a  service  is  personal 
or  not,  that  is  only  half  of  the  problem.  The  other  half  that  needs  to  be  considered  is 
how  the  contract  will  be  administered.  The  best  written  performance  work  statement 
for  non-personal  services  may  be  converted  to  personal  services  by  the  w^  the  con¬ 
tract  is  administered.  As  indicated  above  in  the  Encore  Management  case,  vdien  the 
government,  during  contract  administration,  directs  or  supervises  contractor  emplcyees 
over  a  sustained  period  of  time,  a  personal  service  contract  m^  be  formed.  According 
to  Steven  Schooner,  at  George  Washington  University  Law  School,  this  has  been  a 
problem  in  another  w^;  “woikforce  reductions  and  outsourcing  pressures  have  conspired 
to  increase  the  government’s  reliance  on  personal  services  contracts,  under  which  the 
government  retains  the  function,  but  the  contractor  employees  staff  the  effort.  Under 
such  contracts  rather  than  using  a  performance-based  approach,  agencies  all  too  often 
merely  purchase  labor”  (Schooner,  2004,  p.  73). 

As  competitive  sourcing  continues,  and  even  accelerates,  it  is  sometimes  easy  to  forget 
that  replacing  government  workers  with  contractors  means  more  service  contracts  and 
good  service  contracts  have  historically  been  hard  to  write  (Schooner,  2004,  p.  70). 
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However,  as  Mr  Schooner  has  also  pointed  out,  “Competitive  sourcing  depends  on  skilled 
professionals  planning,  competing,  awarding,  and  managing  sophisticated  long  term  service 
contracts.  But  despite  mandates  to  contract  out  government  functions,  the  administration 
placed  no  concurrent  emphasis  on  retaining  or  obtaining  suitable  acquisition  personnel” 
(Schooner,  2004,  p.  73).  In  fact,  at  the  same  time  agencies  are  being  told  to  complete 
more  jobs  they  are  also  being  told  to  use  performance-based  contracting  for  acquiring 
these  services  with  no  subsequent  increase  in  training  or  personnel. 


The  prindpal  objettive  of  PBSA  is  fo  state  the 
government's  requirements  in  terms  of  performante 
ohieitives  or  ouUomes  desired  rather  than  the 
method  to  he  asm!  in  performante. 


The  principal  objective  of  PBSA  is  to  state  the  government’s  requirements  in  terms 
of  performance  objectives  or  outcomes  desired  rather  than  the  method  to  be  used 
in  performance.  The  problem  here  is  that  this  is  easier  said  than  done.  Acquisition 
personnel  need  experience  and  training  in  order  to  write  effective  PWSs,  and  many 
times  a  team  may  only  work  on  one  A-76  study  in  their  career.  In  a  1999  study 
entitled,  A  Plan  to  Accelerate  the  Transition  to  Performance-based  Services,  Frank 
Anderson,  now  President  of  the  Defense  Acquisition  University  (DAU),  recognized 
that,  “All  participants  in  service  acquisitions  including  the  requirements  developers, 
the  contracting  personnel,  the  program  manager  and  the  quality  assurance  and  contract 
administration  personnel  should  be  involved  in  team  structured  training”  (Anderson, 
1999,  p.  47).  Training  may  indeed  be  the  l^y  to  successful  performance-based  ser¬ 
vice  contracting,  and  the  earlier  in  the  process  they  get  this  training  the  better.  Both 
DAU  and  the  Army  Logistics  Management  Collie  (ALMC)  now  offer  training  in 
PBSA.  To  find  out  more  about  this  training  you  can  go  their  Web  sites  at  www.dau.mil 
or  www.almc.army.mil.  Other  government  agencies,  the  National  Contract  Manage¬ 
ment  Association  (NCMA)  and  a  number  of  private  somces  also  offer  training  in 
PBSA. 

The  PWS  is  the  foundation  on  which  effective  and  efficient  contract  performance 
is  built,  and  writing  a  good  one  is  not  easy.  Training  and  experience  can  obviously 
help  in  this  process,  and  there  are  a  number  of  online  resources  as  well  as  formal 
classes  available.  Here  are  some  suggestions  talsn  from  the  Defense  Logistics  Agency’s 
(DLA)  Commercial  Activities  Guidebook  The  PWS  should  describe  all  requirements 
that  must  be  met  in  clear  concise  wording.  Key  elements  of  the  a  PWS  include  a 
statement  of  the  required  services,  where  performance  is  to  take  place,  the  period  of 
performance,  measurable  performance  standards,  and  the  acceptable  quality  level  or 
allowable  error  rate  (Defense  Logistics  Agency  [DLA],  2004,  pp.  5-4,  5-5).  The  PWS 
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will  become  Section  C  of  the  solicitation  and  should  also  include  general  informa¬ 
tion  the  contractor  needs  to  know,  Government  Furnished  Property  (GFP)  or  services 
that  will  be  provided,  and  quality  control  requirements.  The  PWS  team  will  also 
develop  a  Quality  Assurance  Surveillance  Plan  (QASP)  that  describes  the  procedures 
the  government  will  use  to  monitor  contract  performance.  The  primary  focus  of  the 
QASP  is  to  ensure  the  service  provider’s  quality  control  program  is  working  and  that 
they  are  meeting  the  requirements  of  the  PWS.  Quality  management  responsibility 
is  on  the  service  provider  not  the  government.  The  government  should  be  primarily 
concerned  with  checking  the  service  provider’s  quality  control  program. 

Although  the  PWS  team  usually  writes  the  PWS,  the  contracting  officer  is  ultimately 
responsible  for  it.  They  should  ensure  that  the  services  are  not  personal,  that  the 
services  can  be  procured  in  the  market,  and  the  contract  can  be  administered  so  that 
government  direction  or  supervision  of  contractor  employees  will  not  occm.  Th^r 
also  should  ensiue  the  itwolvement  of  the  PWS  team  including  the  requiring  activity 
and  the  competitive  sourcing  official  in  the  final  review  of  the  PWS  (DLA  2004 
p.5-5,  5-6). 

With  the  confluence  of  PBSA  and  the  new  A-76  competitive  sourcing  initiatives, 
the  need  for  team  training  is  more  important  than  ever  to  insure  that  the  objectives 
of  one  program  do  not  override  the  requirements  of  the  other.  It  is  also  important 
to  understand,  as  stated  in  step  seven  of  Seven  Steps  to  Performance-based  Services 
Acquisition,  “there  is  a  growing  realization  that  the  ‘real  work’  of  acquisition  is  in 
contract  management”  (Office  of  Federal  Procurement  Policy  [OFPP],  2004,  p.  37). 
In  service  acquisition,  contract  management  is  a  “mission  critical  agency  function” 
(OFPP,  2004,  p.  37). 

For  this  reason,  the  acquisition  team  should  be  organized  early,  represent  all 
interested  parties,  be  trained  and  motivated  to  have  a  successful  contract.  Although 
it  is  hard  for  contracting  people  to  swallow  sometimes,  there  is  some  truth  to  the 
statement  that,  “Contract  award  is  not  the  measure  of  success  or  even  an  espe¬ 
cially  meaningful  metric”  (OFPP,  2004,  p.  38).  Effective  and  efficient  contract 
performance  is  the  true  measure  of  a  successful  contract — and  this  success,  to 
a  large  extent,  depends  on  the  experience,  training,  and  motivation  of  the  acqui¬ 
sition  workforce. 


Edward  Allen  Friar  is  a  Professor  of  Contracting  at  the  Defense 
Acquisition  University-South  in  Huntsville,  AL.  Friar  has  over  1 5  years 
contracting  experience  with  the  US  Army  including  the  US.  Army 
Aviation  and  Missile  Commond  at  Redstone  Arsenal  in  Huntsville.  Mr. 
Friar  received  his  master's  degree  in  Public  Administrotion  (MPA) 
from  Arkansas  State  University  and  his  bachelor's  degree  in  fbliticol 
Science  from  Southwest  Missouri  State  University. 

(E-moil  address:  allen.friar@dau.mil} 
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CONTRACT 
ADMINISTRATION  IN  A 
PERFORAAANCE-BASED 
ACQUISITION 
ENVIRONMENT  IS 
SERIOUS  BUSINESS 


JOHN  CAVADIAS 


Many  of  us  are  eager  to  implement  Performance- Based  Services  Acquisitions 
(PBSA)  as  we  attempt  to  comply  with  procurement  and  acquisition  reform. 
Although  there  is  an  abundance  of  written  material  instructing  us  on 
how  to  develop  and  award  a  PBSA,  we  find  far  less  guidance  on  the 
emerging  realities  in  administering  an  awarded  PBSA.  Contract  adminis¬ 
tration  in  a  PBSA  environment  is  mission-critical,  not  to  be  treated 
as  an  ancillary  responsibility  subordinate  to  originating  acquisitions. 
This  article  approaches  this  viewpoint  by  examining  the  post-award 
management  of  a  single  service  in  one  commercial  industry  and  compares 
it  to  government  contracting  practices  particularly  with  an  emphasis  on 
legacy  cradle-to -grave  organizational  structures  while  also  exploring 
the  need  for  a  shift  in  government  perspective  and  change  in 
organizational  practices. 


increasing  the  use  of  Performance-based  Services  Acquisitions  (PBSA)  to  40 
percent  of  eligible  service  actions  over  $25,000,  during  fiscal  year  2005,  is  the 
target  achievement  level  for  all  federal  government  contracting  offices  procuring 
services  (Office  of  Management  and  Budget  [0MB],  2004).  Regardless  of  vdiat  agency, 
which  contracting  command,  or  the  size  of  the  contracting  office,  acquisition  reform 
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applies  to  all  those  in  the  business  of  procuring  and  administering  services  in  the 
federal  government.  As  agencies  begin  to  develop  their  management  plans  for 
the  Office  of  Management  and  Budget  (0MB)  outlining  their  approach  to  increase 
the  use  of  PBSA,  it  is  crucial  that  agencies  give  considerable  attention  to  managing 
the  post-award  contract  administration  phases  just  as  they  will  do  so  with  pre-award 
planning. 

Several  military  components  within  the  Department  of  Defense  (DoD)  pass  their 
contract  administration  functions  onto  the  Defense  Contract  Management  Agency 
(DCMA)  for  post-award  execution.  Not  all  contracting  offices,  however,  can  assign 
their  contracts  to  the  DCMA  for  contract  administration.  It  is  clear  that  the  DCMA 
does  not  engage  in  garrison  contract  management.  The  responsibility  of  that  function 
usually  falls  to  the  installation  or  tenant  commander  if  the  contract  is  for  a  base,  post, 
camp,  or  station  (Defense  Federal  Acquisition  Regulation  Supplement  [DFARS],  2001). 
Moreover,  the  DCMA  must  now  focus  their  attention  and  limited  resources  on  major 
program  efforts  and  critical  readiness  items.  Accordingly,  even  a  billion-dollar  PBSA 
such  as  the  Marine  Corps’  (USMC)  garrison  food  service  contract  is  managed  at  the 
installation  level  since  it  is  outside  the  mission  of  DCMA  (Defense  Contract 
Management  Agency  [DCMA],  n.d.). 

Our  starting  point  for  discussion  begins  here,  at  home  base,  vv^iere  we  originate  our 
PBSA  contracts.  After  awarding  a  PBSA,  many  of  us  must  also  face  the  challenges 
of  administering  a  contract  that  demands  specialized  skills  and  resources  beyond  simple 
contract  compliance  (Interagency-Industry  Partnership  in  Performance,  2003). 
Managing  performance  in  a  PBSA  environment  is  a  whole  new  ball  game  in  the 
world  of  government  contract  administration. 


WHO'S  ON  FIRST? 

At  contracting  offices  that  manage  contracts  from  inception  to  closeout — or  cradle- 
to-grave — such  as  those  contracts  for  base,  post,  camp  and  station  on  military  instal¬ 
lations,  the  unwritten  prioritized  order  of  work  is  typically  (1)  pre-award  procure¬ 
ment/acquisition,  (2)  post-award  crisis  management,  and  lastly,  (3)  post-award  contact 
administration.  In  these  contracting  shops,  contract  administration  is  given  the  lowest 
priority  and  is  sometimes  viewed  as  the  necessary  evil  of  the  business.  This  is  greatly 
so  because  the  principal  metrics  classical^  established,  in  order  of  importance,  consist 
of  (1)  contract  dollars  procured,  (2)  number  of  contract  awards,  and  (3)  procurement 
administrative  lead  time  (PALT),  which  is  the  total  time  it  takes  to  award  a  contract — 
the  quieter  the  better.  Conversely,  post-award  contract  administration  contributes  almost 
nothing  to  these  three  guiding  metrics;  therefore,  it  is  difficult  to  justify  an  invest¬ 
ment  of  labor  into  a  post-award  operation  that  does  not  point  to  a  visible  cost-benefit 
with  regard  to  these  conventional  metrics. 

This  outdated  logic  can  no  longer  add  any  value  to  maintaining  an  effective  program 
when  feced  with  managing  PBSA.  We  find  that  PBSA  introduces  a  host  of  additional 
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post-award  metrics  seldom  seen  within  traditional  non-PBSA  contracting.  The  standards 
of  measurement  in  a  PBS  A  become  more  results-oriented  such  as  quality  of  work  or 
product,  accessibility,  timeliness,  accuracy,  and  customer  satisfaction.  Once  perfor¬ 
mance  metrics  are  defined  and  methodicalty^  developed  in  the  contract  formation  phase, 
the  acquisition  team  cannot  comfortably  disperse  immediately  after  contract  award  in 
the  assumption  that  first-rate  contract  performance  will  consistently  materialize 
according  to  plan  throughout  the  execution  phase.  On  the  contrary,  each  member  of 
the  team  must  tirelessly  work  together  managing  performance  from  the  point  of  contract 
award  until  contract  closeout. 


Hlony  €ommertial  industries  have  long  retognized 
that  the  servmng  or  administration  arm  of  their 
husinesses  is  as  mission-aithai  as  their 
origination  or  produttion  stage. 


PBSA  requires  a  level  of  commitment  and  teamwork  from  all  involved  that  exceeds 
that  required  in  other  types  of  contracts  (Know  Net,  n.d.).  There  is  no  escaping  this 
new  reality.  The  success  of  a  PBSA  is  highly  dependent  on  the  effort  and  resources 
invested  into  monitoring  performance  by  using  many  sophisticated  tools  and  metrics 
including  performance  indicators  and  standards,  quality  assurance  surveillance  plans 
(QASP),  performance  requirements  summaries  (PRS),  acceptable  quality  levels  (AQL), 
and  appropriate  positive  as  well  as  negative  incentives.  The  question  is:  With  essential 
post-award  metrics  that  carry  such  heavy  consequence,  are  most  cradle-to-grave 
contracting  offices  equipped  with  the  resources  to  manage  PBSA  in  a  political 
environment  of  federal  budget  constraints  and  acquisition  workforce  downsizing?  More 
important,  even  if  supplied  with  adequate  resources,  will  the  three  principal  metrics 
continually  eclipse  the  importance  of  improving  the  PBSA  post-award  metrics?  It 
seems  that  agencies  focus  their  attention  on  awarding  contracts,  not  on  managing 
them  once  awarded  (Schooner,  2004). 

Today  in  this  burgeoning  PBSA  environment,  the  agencies’  needs  for  contract 
oversight  are  expanding,  not  decreasing.  Thus,  contract  performance  management 
urgently  requires  an  infusion  of  investment  into  post-award  contract  administration 
operations.  Many  commercial  industries  have  long  recognized  that  the  servicing  or 
administration  arm  of  their  businesses  is  as  mission-critical  as  their  origination  or 
production  stage.  This  article  includes  discussions  of  one  such  industry  to  illustrate 
the  vital  importance  of  managing  a  business  with  adequate  and  specialized  resource 
allocation  among  all  operational  phases  and  the  consequences  if  neglected. 
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ACQUISITION  BEViEW  IDiigMAI 


WE  CAN  ALL  LEARN  FROM  INDUSTRY 

The  Under  Secretary  of  Defense  had  stated,  “PBSA  strategies  strive  to  adopt  the 
best  commercial  practices... to  achieve  greater  savings  and  efficiencies”  (Department 
of  Defense  [DoD]  Acquisition  and  Technology,  2001,  p.  1).  We  can  safely  presume 
that  this  announcement  not  only  applies  to  the  acquisition  planning,  contract  forma¬ 
tion,  and  source  selection  phases,  but  also  to  the  execution  management  phases  of  a 
PBSA. 

Contract  administration  is  rather  analogous  to  the  loan  servicing  operation  within 
the  mortgage  banking  business,  a  commercial  industry  to  which  most  homeowners 
can  relate.  We  might  be  able  to  use  some  of  the  same  lessons  from  the  mortgage 
business  as  we  view  the  business  of  government  contract  administration.  To  illustrate 
this  analogy,  let  us  follow  the  phases  of  how  a  loan  is  originated  and  subsequent^ 
administered  once  the  loan  is  funded,  a  process  that  is  similar  to  procuring  and  ad¬ 
ministering  a  contract. 

When  a  financial  lending  organization  receives  a  requirement  for  a  mortgage  loan, 
it  goes  through  a  phase  akin  to  tasks  performed  in  government  acquisition  planning. 
The  processing  of  a  loan  will  include  completing  a  requirements  analysis;  credit 
analysis,  employment  verifications,  and  property  appraisals;  methods  of  financing; 
and  pricing  arrangements.  In  government  contracting,  these  tasks  are  respectively 
parallel  to  developing  an  acquisition  plan  and  performance  work  statement;  past 
performance  review;  determining  contractor  responsibility;  contracting  methods;  and 
choosing  a  contract  type.  Once  completed,  the  processed  loan  package  will  progress 
onto  a  warranted  loan  officer  for  underwriting,  not  unlike  the  contract  formation 
phase,  where  the  pre-award  package  undergoes  a  final  review  to  ensure  it  conforms 
to  all  required  guidelines. 


Control  administration  is  rather  analogous  to  the 
loan  servking  operation  within  the  mortgage 
hanking  business,  a  rommeriiai  industry  to  whhh 
most  homeowners  tan  relate. 


Finally,  the  loan  package  is  closed,  official  documents  are  drawn,  and  the  loan  is 
awarded  and  funded  to  the  holder  of  the  new  mortgage  (borrower).  Throughout  this 
process,  a  lender  must  abide  by  hundreds  of  statutes,  relations,  and  guidelines, 
similar  to  government  contracting  under  the  Federal  Acquisition  Relations  (FAR) 
and  its  supplements.  There  are  also  socioeconomic  considerations  the  lender  must 
heed  throughout  all  phases.  In  the  lending  business,  these  pre-funding  stages  are 
referred  to  as  loan  origination. 
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Once  a  lender  completes  the  loan  origination  stage,  the  funded  mortgage  loan  is 
usually  sold  to  investors  via  the  secondary  markets,  e.g.,  Ginnie  Mae.  Most  borrowers 
never  realize  that  their  loan  contract  is  sold  to  outside  investors  since  the  transaction 
is  transparent.  For  the  government,  they  too  have  their  own  group  of  investors  who 
fund  acquisitions — ^the  American  taxp^er. 

The  final  and  longest  phase  in  the  life  cycle  of  a  loan  contract  is  known  as  loan 
servicing  where  it  is  administered.  Lenders  have  the  choice  of  either  outsourcing  this 
part  of  the  operation  or  retaining  the  servicing  operation  in-house.  Regardless  of 
whether  servicing  is  released  or  retained  with  the  original  lender,  this  subsequent 
post-funding  stage  is  where  the  real  work  begins.  Here,  dozens  of  crucial  adminis¬ 
tration  tasks  must  be  implemented  and  monitored  throughout  the  life  of  the  loan 
contract. 


For  the  government,  they  too  have  their  own 
group  of  investors  who  fund  arguisitions-^ 
the  Ameriran  taxpayer. 


Loan  origination  might  have  taken  only  a  matter  of  weeks  or  a  few  months  from 
requisition  to  award.  However,  in  the  succeeding  loan-servicing  phase,  a  single  loan 
contract  requires  years  of  administration — five  to  seven  on  average  but  can  go  beyond 
30.  In  this  multifaceted  administration  phase,  a  collaborative  team  of  skilled  profes¬ 
sionals  engages  in  a  form  of  daify  performance  monitoring  adhering  to  a  clear  set  of 
standards  and  metrics.  The  loan-servicing  workforce  required  to  run  such  a  post¬ 
award-like  operation  is  sizeable,  diverse,  and  specialized.  Their  positions  include 
customer  service  representatives;  quality  assurance  specialists;  escrow/impound 
analysts;  government  compliance  and  reporting  specialists;  risk  investigators;  debt 
collectors;  default  appraisal  managers;  bankmptcy  and  foreclosure  specialists;  and 
investor  accountants,  among  others. 


WE  CAN  ALL  SUFFER  FROM  POOR  ADMINISTRATION 

Can  you  imagine  the  harmful  consequences  to  the  lender  if  insufficient,  scattered, 
or  unspecialized  resources  were  assigned  to  managing  the  post-funding  phase  of 
lending?  Pferhaps  a  borrower  would  discover  that  his  property  tax  bill  was  paid  late 
through  negligence  by  the  loan  servicer,  resulting  in  losing  their  property  through  a 
tax  sale.  How  might  the  financial  fate  of  the  investment  in  the  loans  change  if  the 
lender  stopped  investing  adequate  resources  toward  executing  vigilant  post-funding 
oversight?  Perhaps  property  insurance  analysts  would  n^lect  to  check  whether 
mandatory  fire  insurance  premiums  are  paid.  If  true,  then  lender’s  collateral 
guaranteeing  the  loan  contract  would  be  at  high  risk  of  total  loss  should  it  bum  down. 
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Other  ensuing  scenarios  might  include  losses  from  abandoning  fiscal  management 
oversight.  Without  properly  monitoring  the  aging  of  mortgage  accounts,  debt  collections 
would  never  begin,  as  no  one  would  know  ^^hich  borrowers  are  delinquent.  In  turn, 
a  high  percentage  of  the  loan  portfolio  would  increasingly  fail  as  revenues  from  monthly 
mortgage  payments  decrease  or  discontinue  all  together.  Delinquent  borrowers  that 
could  have  been  salvaged  by  debt  collectors  at  the  early  stages  of  default  via  informal 
remedies  now  would  find  themselves  beyond  the  point  of  recovery.  Homes  that  should 
have  been  foreclosed  long  ago  would  turn  into  rent-free  dwellings.  Private  and  public 
investors  in  these  now  deteriorating  mortgage  portfolios  would  risk  losing  a  consid¬ 
erable  portion  of  their  financial  investment.  In  short,  without  the  essential  resources 
fully  employed  in  post-award  administration,  results  would  turn  disastrous  for  all  the 
parties. 


Poor  performante  monitoring,  sporndU  quniity 
assuranre,  little  effort  expended  into  managing 
ihanges  or  settling  disputes  does  indeed  result  in 
damages  to  both  government  and  rontrattor 
as  it  does  with  produter  and  ronsumer. 


Similar  consequences  can  occur  with  inadequate  resources  in  government  contract 
administration.  Poor  performance  monitoring,  sporadic  quality  assurance,  little  effort 
expended  into  managing  changes  or  settling  disputes  does  indeed  result  in  damages 
to  both  government  and  contractor  as  it  does  with  producer  and  consumer.  Undesir¬ 
able  outcomes  from  the  deficient  employment  of  suitable  and  adequate  resources 
become  highfy  magnified  and  perilous  in  a  PBSA  environment.  Let  us  now  look  at 
a  typical  contracting  office,  which  is  extra  vulnerable  to  finding  themselves  chal¬ 
lenged  with  PBSA  contract  administration,  such  as  those  offices  at  the  military  in¬ 
stallation  level  that  are  unsupported  by  the  DCMA. 


WHArS  ON  SECOND? 

In  the  traditional  world  of  non-performance-based  contracting,  many  awarded  service 
contracts  may  appear  fully  automated  during  its  post-award  life  cycle.  Some  believe 
there  is  no  need  to  shift  gears  in  this  post-award  phase  because  it  is  perceived  as  self- 
relating.  The  contractor  delivers  the  service,  perhaps  with  only  modest  issues  or 
few  problems  along  the  way.  Contract  specialists  can  often  get  aw^  with  ignoring 
many  of  their  post-awarded  contracts,  with  little  consequence,  as  they  work  on  new 
procurements  until,  for  example,  an  option  to  extend  services  comes  due. 
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On  the  surface,  ignoring  most  administration  duties  usually  goes  unnoticed.  However, 
when  a  potentially  serious  problem  does  crop  up  requiring  immediate  action  on  the 
part  of  the  contracting  office,  the  situation  quickly  becomes  a  firefight.  Fighting  fires 
is  second  in  order  of  prioritized  jobs  previously  referred  to  as  post~award  crises 
management.  All  business  in  the  pre-award  arena  must  come  to  a  halt  as  energy  now 
focuses  on  extinguishing  the  crises  at  hand.  Once  the  fire  is  ostensibly  extinguished, 
it  is  back  to  originating  new  contracts  although  sizzling  embers  from  those  earlier 
fires  may  still  bum  beneath  the  surface,  perhaps  waiting  to  re-ignite. 

While  the  contract  administration  file  often  sits  in  a  remote  file  cabinet,  out  of 
sight,  out  of  mind  becomes  more  of  a  standard  practice  than  just  a  figure  of  speech. 
Though  supervisory  contract  specialists  practically  never  encourage  this  behavior,  it 
is  frequently  condoned.  The  underlying  reasons  are  internal^  evident:  metrics.  The 
contract  specialist  has  little  time  to  perform  duties  perceived  as  non-critical  while 
continually  pressured  to  rapidly  originate  more  contracts  in  order  to  increase  the 
contract  dollars  procured  and  number  of  contracts  awarded.  Moreover,  new 
requirements  must  be  awarded  expeditiously  in  accordance  with  another  significant 
metric  mentioned  earlier,  PALT.  These  are  the  metrics  most  often  monitored  and 
evaluated  by  government  leaders  from  supervisory  contract  specialists  all  the  w^  up 
to  congressional  representatives. 

Considering  the  exposure  these  metrics  receive,  it  is  no  wonder  that  government 
investment  into  pre-award  contracting  activities  rises  above  other  competing  priorities, 
most  detrimentally  above  contract  administration  that  appear  to  carry  no  worthy  metrics. 
However,  there  is  one  worthy  metric  in  contract  administration  that  overridingly  seems 
to  capture  the  attention  at  the  highest  levels  of  government:  the  media  broadcasting 
of  embarrassing  news  stories  whenever  contracts  fail  to  self-regulate. 


Fighting  fires  is  setond  in  order  of 
prioritized  fobs  previously  referred  to 
as  post-award  crises  maaagemenf. 


The  oiganizational  views  on  contract  administration  in  this  cradle-to-grave  structure 
appear  to  murmur.  No  more  time  than  is  absolutely  required  should  be  devoted  to  this 
less  visible  job.  So  in  effect,  time  invested  administering  contracts  equate  to  time  lost 
procuring  new  purchase  requirements.  Therefore,  we  can  infer  that  performing  con¬ 
tract  administration  in  this  cradle-to-grave  environment  emerges  mostly  as  a  distraction 
and  a  hindrance  to  the  primary  goals  of  the  contracting  office,  which  are  to  procure 
more  contracts  to  fulfill  those  highly  visible  metrics. 

Occasionally,  many  of  us  do  come  upon  some  highly  talented  contract  specialists 
that  are  quite  masterful  at  juggling  both  procurement  and  administration  tasks. 
Recruiting  the  finest  jugglers  into  government  contracting,  however,  is  not  a  long-teim 
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or  viable  solution.  The  fundamental  dilemma  does  not  lie  within  uncovering  and 
developing  the  multi-tasking  skills  of  the  individual  contract  specialist.  The  proMem 
is  with  continuing  to  expect  optimum  results  in  both  procurement  and  administration 
from  a  single,  group,  or  department  of  contract  specialists  whose  behavior  is  almost 
entirely  directed  by  three  metrics— dollars  procured,  number  of  awards,  and  PALT. 
The  mind-set  that  we  can  do  it  all  as  we  transition  toward  PBSA  contract  manage¬ 
ment  must  be  seriously  reconsidered  if  we  are  to  ensure  the  highest  possible  resource 
utilization  and  level  of  success  in  this  emerging  world  of  performance-based  contract 
management. 


MANAGE  RESOURCE  ALLOCATION  FOR  OPTIMUM  RESULTS 

Ask  yourself  as  you  think  back  to  the  loan  servicing  scenarios  presented  earlier: 
Can  the  loan  originators  do  it  all?  Can  they  successful^  monitor  a  diverse  portfolio 
of  post-awarded  loans  as  they  juggle  loan  origination  efforts  to  satisfy  prospective 
homeowners’  expectations  for  a  quick  loan  closing?  Furthermore,  can  the  debt 
collectors  simultaneously  meet  the  demands  for  originating  new  loans  as  they  juggle 
their  loan  servicing  efforts  toward  meeting  aggressive  low-delinquency  goals  to  protect 
investors?  Most  would  agree  it  seems  unreasonable  to  expect  that  such  a  misallocation 
of  human  resources  could  produce  the  desired  objectives  in  either  the  loan  origination 
phase  or  loan-servicing  phase  of  a  lending  operation. 


Classkal  etonomU  thinking  eniourages 
intrensing  the  division  of  iabor  to 
improve  effUienty  and  growth. 


Classical  economic  thinking  encourages  increasing  the  division  of  labor  to  improve 
efficiency  and  growdh.  Therefore,  it  should  be  recognized  by  and  adapted  to  contract 
administration  as  has  always  been  standard  practice  in  commercial  industry.  Invest  in 
sufficient  human  resources  and  specialize  the  workforce  to  the  job  requirements 
considering  the  entire  organizational  structure  and  its  goals. 


CLOSING  REMARKS 

Managing  contracts  in  today’s  PBSA  environment  demands  leadership’s  unwavering 
commitment  to  build  a  contracting  organization  that  will  not  find  their  contracting 
professionals  continually  divided  between  chairing  a  source  selection  committee  and 
negotiating  multi-million  dollar  equitable  adjustments.  One  priority  will  forever  battle 
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to  overtake  another  higher,  equivalent,  or  lesser  priority.  Each  phase  of  contracting 
is  justifiably  and  distinct^  a  mission-critical  entity  that  warrants  specialized  resources. 
These  entities  must  equally  coexist  so  that  competing  priorities  no  longer  struggle  to 
survive  at  the  same  time,  in  the  same  space.  Preserving  a  cradle-to-grave  operation 
under  a  single  edict  of  three  metrics  will  only  guarantee  an  early  grave  for  acquisition 
reform. 

If  we  are  to  ensure  that  government  and  contractor  both  fulfill  their  contractual 
obli^tions  under  a  PBSA,  we  must  invest  in  the  right  mix  of  specialized  talent  as 
we  undertake  these  aggressive  acquisition  reforms.  Plan  for,  invest  in,  and  fight  for 
all  the  right  resources  to  make  your  PBSA  a  total  success  throughout  the  entire  life 
of  the  acquisition.  Invest  early,  discriminately,  and  prudently,  so  that  we  can  guarantee 
that  our  ultimate  investors,  the  American  taxp^ers,  will  continue  to  reap  a  meaningful 
return  on  their  investment  in  the  most  productive  government  in  the  world. 


John  Covodias  is  the  contracting  officer  for  the  largest  Fferformance- 
Based  Services  Acquisition  in  the  U.S.  Marine  Corps  (USMC).  Prior  to 
this  assignment,  he  led  formal  contracting  for  USMC's  Regional 
Contracting  Office  Southwest,  which  performs  cradle-to-grave 
contracting  He  holds  an  M.BA.  from  California  State  University,  San 
Marcos;  a  B.BA  in  finance/banking;  and  recently  was  recognized  by 
Assistant  Chief  of  Staff  (AC/S)  Logistics  as  Employee  of  the  Year  for 
organizing  the  new  contract  administration  team  at  Camp  Pendleton. 

(E-mail  address:  john.cavadias@usmc.mil) 
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TEST  AND  EVALUATION 
IN  A  DYNAMIC 
ACQUISITION 
ENVIRONMENT 

GREGORY  L  BARNETTE 


Acquisition  refornn  and  the  implementation  of  agile  acquisition  processes 
within  the  Air  Force  are  allowing  acquisition  professionals  greater  flexibility 
in  meeting  user  requirements.  A  strong  emphasis  is  being  placed  on  using 
mature  systems  and  technologies,  allowing  new  programs  to  be  Initiated  at 
any  point  in  the  acquisition  process  continuum.  Changes  have  been  incor¬ 
porated  into  the  test  and  evaluation  (T&E)  process  to  support  the  increased 
flexibility.  Judicious  use  of  the  various  types  of  recognized  tests  allow  the 
program  manager  (PM)  to  reduce  risk  and  ensure  performance  expecta¬ 
tions  are  met.  The  following  provides  an  overview  of  the  test  tools  available 
to  the  acquisition  professional  and  highlights  the  evolutionary  changes  that 
were  recently  Incorporated  into  Air  Force  test  guidance. 


Headquarters  Air  Force/Test  and  Evaluation  (HQ  AF/TE)  released  a  new  test 
and  evaluation  instruction  (Air  Force  Instruction  [AFI]  99-103)  on  August  6, 
2004.  The  release  of  AFI  99-103  corresponded  with  a  rewrite  of  AFI  63-101, 
Capabilities  Based  Acquisition,  April  1,  2004  (interim  approval)  and  AFI  10-601, 
Capabilities  Based  Requirements  Development,  July  30,  2004,  bringing  all  the  instructions 
in  line  with  direction  found  in  Chairman  Joint  Chiefs  of  Staff  Instructions  (CJCSIs) 
3170.01D  (2003),  Joint  Capabilities  Integration  and  Development  System,  and  6212.01C, 
Interoperability  and  Supportdbility  of  Information  Technology  and  Nation  Security  Sys¬ 
tems,  Perhaps  more  important^,  the  new  AFI  99-103  (2004)  incorporated  evolutionary 
changes,  reflecting  the  test  and  e\aluation  community’s  adaptations  to  the  Air  Force’s 
implementation  of  the  agile  acquisition  processes  over  the  last  decade.  The  instmc- 
tion  also  poses  a  seamless  verification  process  that  fosters  an  integrated  testing 
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philosophy  in  an  effort  to  streamline  Test  and  Evaluation  (T&E),  much  as  the  ac¬ 
quisition  process  itself  is  being  streamlined. 

This  effort  to  integrate  testing  is  born  out  by  the  fact  that  the  new  API  99-103 
(2004)  is  itself  a  compilation  of  the  former  API  99-101  (1996),  Developmental  Test 
and  Evaluation-,  API  99-102  (1998),  Operational  Test  and  Evaluation-,  and  API  99- 
105  (1994),  Live  Fire  Test  and  Evaluation.  The  following  shows  how  the  traditional 
T&E  process  supports  the  spiral  acquisition  philosophy,  describes  how  the  types  of 
tests  support  the  new  acquisition  philosophy,  and  highlights  new  test  programs  that 
facilitate  technology  transition  to  the  warfighter. 


TRADITIONAL  TEST  AND  EVALUATION  SUPPORT 
FOR  SPIRAL  ACQUISITION 

Though  the  acquisition  process  has  evolved  along  with  the  county  itself,  the  stucture 
for  much  of  the  modem  acquisition  processes  was  put  in  place  by  the  McNamara  re¬ 
forms  and  remains  fundamentally  unchanged.  This  was  done  to  reign  in  a  system  that 
was  perceived  as  out-of-control,  as  evidenced  ly  aircraft  cost  overruns  that  were  as  much 
as  100  percent  after  inflation  adjustment  toward  the  end  of  the  Korean  War  (Harman, 
1997).  Standardized  procedures  and  processes  replaced  those  developed  by  services  and 
individual  program  offices.  Programs,  especially  major  systems  procurements,  were 
expected  to  adhere  rigorously  to  this  process  (shown  in  Figure  1).  To  progress  through 
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FIGURE  1. 

ACQUISITION  PROCESS  WITH  TEST  AND  EVALUATION  TIMELINES 


the  process,  a  program  had  to  pass  through  a  series  of  well-defined  g^tes,  designed 
to  minimize  risk  to  the  government.  The  pregram  had  to  demonstrate  its  maturity  by 
passing  a  series  of  tests  at  each  ^te  that  were  designed  to  ensure  the  development  was 
progressing  adequately.  The  tests  were  on  a  continuum  that  focused  initially  on  the 
engineering  aspects  of  the  program  and  shifted  later  to  operational  concerns. 

Requirements  under  United  States  Code  Title  10  §2399  (2002)  drew  a  clear  distinc¬ 
tion  between  the  two  types  of  testing  hy  requiring  the  user  to  demonstrate  the  effec¬ 
tiveness  of  the  final  design  in  an  operational  environment.  This  further  dichotomized 
developmental  test  and  evaluation  ^T&E),  where  the  acquisition  community  tested 
systems  and  subsystems  against  engineering  and  contract  specifications,  and  opera¬ 
tional  test  and  evaluation  (OT&E),  where  operators  tested  the  entire  systems  including 
its  support  infrastructure  against  the  mission  requirements.  This  dichotomy  was  em¬ 
phasized  by  the  requirement  for  the  DT&E  test  organization  to  certify  the  system  as 
ready  for  operational  testing  (see  AFMAN  63-119). 


The  DT&E  portion  of  the  test  and  evaluation 
process  has  been  inherently  more  flexible  by 
virtue  of  having  the  requirements  developed 
internally  to  the  program  offUe  and  tan 
torrespondingly  adapt  to  the  requirements 
of  a  spiral  atquisition  professes. 
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The  DT&E  portion  of  the  test  and  evaluation  process  has  been  inherently 
more  flexible  by  virtue  of  having  the  requirements  developed  internally  to  the 
program  office  and  can  correspondingly  adapt  to  the  requirements  of  a  spiral 
acquisition  processes.  The  program  office’s  design  integrated  product  team  (IPT) 
is  delivered  a  set  of  mission  requirements  for  the  program,  along  with  any  as¬ 
sociated  key  performance  parameters  (KPPs).  The  design  IPT  takes  the  mission- 
level  requirements  and  develops  system  design  requirements  and  specifications 
that  are  traceable  to  the  KPPs.  The  program  office  now  has  the  latitude  to  select 
which  design  requirements  and  specifications  are  critical  technical  parameters 
(CTPs),  subject  to  the  milestone  decision  authority  (MDA)  approval  (and  Office 
of  the  Secretary  of  Defense  [OSD]  approval  for  programs  on  the  oversight  list). 

An  example  of  a  KPP  and  its  derived  CTPs  could  be  the  user  requirement  for 
supersonic  cruise  capability  in  military  power  and  the  respective  engineering 
requirements  for  a  maximum  take-off  weight  and  minimum  thrust  to  accomplish 
this  cruise  capability.  The  CTPs  normally  drive  the  DT&E  test  requirements  and 
the  program  office  gets  to  decide  what  level  and  degree  of  testing  is  required. 
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If  the  contractor  test  is  deemed  adequate,  data/report  review,  over-the-shoulder 
observation,  or  test  participation  may  be  the  extent  of  government  involvement.  If 
government  testing  is  required,  a  DT&E  agency  will  be  given  the  requirements  to 
execute  at  decision  points  within  the  development  efFort.  Since  the  program  office 
owns  the  design  process,  chooses  which  parameters  to  track/test,  and  controls  what 
pes  to  the  DT&E  process,  their  trade  space  in  making  those  decisions  includes 
impacts  due  to  the  DT&E  test  requirements. 

The  two  biggest  changes  to  the  DT&E  process  stem  from  the  integrated  test  approach 
espoused  in  API  99-103  (2004)  and  the  preference  to  employ  combined  DT&E  and 
OT&E,  whenever  practical.  Collaboration  between  the  designated  OT&E  and  DT&E 
agencies  has  been  historically  focused  on  the  development  of  the  test  and  evaluation 
master  plan  (TEMP)  and  preparation  for  certification  for  dedicated  OT&E.  The  new 
requirement  to  establish  an  integrated  test  team  (ITT)  at  the  onset  of  a  program  wdll 
ensure  contractor,  developmental,  and  operational  testers  remaining  actively  en¬ 
gaged  throughout  the  program.  This  will  assist  in  obtaining  the  goal  of  seamless 
verificatioii  by  helping  to  eliminate  duplicate  test  requirements,  correcting 
scheduling  inefficiencies,  identifying  performance  concerns  early,  and  smooth¬ 
ing  the  transition  to  dedicated  OT&E.  An  ITT  will  also  facilitate  the  accomplish¬ 
ment  of  combined  DT&E  and  OT&E  by  ensuring  OT&E  requirements  are  commu¬ 
nicated  early  and  incorporated  into  DT&E  planning. 


Numerous  failings  of  frithal  operation  issues 
(COIs)  during  operational  testing  have  resulted  in 
detertifying  the  systems  and  sending  them  hath 
into  development  to  <orre<t  the  defitiendes. 


OT&E  test  agencies  have  traditionally  become  heavily  involved  in  active  testing 
following  certification  for  dedicated  OT&E.  Numerous  failings  of  critical  operation 
issues  (COIs)  during  operational  testing  have  resulted  in  decertifying  the  systems 
and  sending  them  back  into  development  to  correct  the  deficiencies.  This  has  prompted 
the  Director,  OT&E  (OSD)  and  HQ  AF/TE  to  push  for  more  active  participation  by 
OT&E  agencies  earlier  in  the  development  process.  This  has  led  to  the  implemen¬ 
tation  of  numerous  types  of  user  demonstrations  that  are  executed  in  parallel  with 
the  development  process,  such  as  Early  Operational  Assessments  (EOAs),  Opera¬ 
tional  Assessments  (OAs),  and  Operational  Utility  Evaluations  (OUEs),  which  are  all 
authorized  under  API  99-103.  Operational  testers  have  also  been  encouraged  to 
participate  in  DT&E  tests  as  observers  to  facilitate  early  communication.  (Note:  The 
legal  requirements  for  operational  testing  are  to  support  production  and  fielding 
decisions  at  the  end  of  the  acquisition  process.) 
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OT&E  has  begun  to  have  a  profound  impact  on  DT&E  by  program  offices 
implementing  combined  DT&E/OT&E  for  cost  and  schedule  efficiencies.  Policies 
implemented  as  the  result  of  the  legal  requirement  for  operational  testing  preclude 
contractor  participation  in  the  execution  of  the  test  (except  as  planned  for  in  the 
concept  of  operations),  data  collection  and  analysis  by  the  prime  contractor,  the  use 
of  prototype  hardware,  use  of  a  non-representative  test  environment,  and  changes 
to  the  test  articles.  None  of  these  prohibitions  apply  to  DT&E,  but  must  be  observed 
during  combined  testing  to  carry  data  forward  into  OT&E  anafysis. 


There  are  fwo  types  of  qualifitation  testing: 
qualifitation  test  and  evaluation  (QT&E), 
analogous  to  DT&E,  and  qualifiration 
operational  test  and  evaluation  (QOT&E), 
analogous  to  Initial  OT&E, 


Air  Force  99-series  instructions  and  relations  have  historicalfy^  allowed  for  the 
performance  of  qualification  testing.  Qualification  testing  was  designed  to  facilitate 
procurement  of  non-developmental  items  (NDI),  ready  to  be  put  to  use  in  the  Air 
Force.  There  are  two  types  of  qualification  testing:  qualification  test  and  evaluation 
(QT&E),  analogous  to  DT&E,  and  qualification  operational  test  and  evaluation 
(QOT&E),  analogous  to  initial  OT&E.  Successful  completion  of  a  QOT&E  provided 
results  for  milestone  C  entry  for  a  non-developmental  item.  The  availability  of  QT&E 
allowed  for  earlier  testing  of  an  NDI  system  that  may  require  minor  modification  or 
perhaps  had  some  issue  concerning  readiness  for  QOT&E.  The  existence  of  quali¬ 
fication  testing  demonstrates  flexibility  in  the  existing  process  to  support  some  of  the 
currently  touted  agile  acquisition  philosophies.  It  just  lacked  the  present  emphasis. 


EVOLUTIONARY  TEST  TOOLS 

One  of  the  primary  goals  of  agile  acquisition  is  to  shorten  the  acquisition  cycle 
time.  This  can  be  accomplished  by  either  reducing  the  time  it  takes  to  accomplish 
the  process  or  by  entering  the  acquisition  cycle  at  a  later  stage  in  the  process.  To 
accomplish  the  later,  OSD  has  implemented  a  hierarchy  of  solutions  for  meeting 
requirements  via  a  new  start  acquisition  program.  This  directs  decision  makers  to 
look  for  solutions  where  development  costs  are  minimal  and  design  solutions  are 
mature.  This  hierarchy  is  listed  below  in  order  of  descending  preference: 
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system  capability.  The  OSD  programs  include  the  Foreign  Comparative  Test  (FCT) 
and  Advanced  Concept  Technology  Demonstration  (ACTD).  The  Air  Force  created 
mission-oriented  battlelabs. 

The  FCT  Program  is  managed  by  the  Comparative  Testing  Office  (CTO)  under 
the  Deputy  Under  Secretary  of  Defense,  Advanced  Systems  &  Concepts,  Office  of 
the  Under  Secretary  of  Defense  (Acquisition,  Technology  &  Logistics).  The  FCT 
program  brings  value  to  the  acquisition  process  by  using  foreign  military  and  com¬ 
mercial  non-developmental  items  to  enter  the  process  at  milestone  B  or  C  and  bypass 
all  or  part  of  the  development  process.  The  FCT  Program  is  authorized  ly  Title  10, 
United  States  Code,  Section  2350a(g)  and  funded  by  Office  of  the  Secretary  of 
Defense  (OSD)  Research,  Development,  Test  and  Evaluation  (RDT&E)  appropria¬ 
tions.  The  objectives  of  the  FCT  program,  as  stated  in  the  handbook,  are  to  improve 
US.  warfighter’s  capabilities  and  reduce  expenditures.  The  handbook  provides  cradle 
to  grave  instruction  for  an  FCT  and  can  be  found  at  the  CTO  s  Web  site  http.// 
www.acq.osd.mil/cto. 


The  purpose  of  an  ACTD  is  to  demonstrate  the 
effertiveness  of  mature  or  maturing  tethnology 
to  meet  a  iritUal  user  need. 


Two  categories  of  FCTs  are  authorized:  procurement  testing  and  technical 
assessment.  Procurement  testing  focuses  on  finding  a  materiel  solution  for  an 
existing  requirement.  Although  technical  assessments  for  the  examination  of  po- 
tentialfy  revolutionary  foreign  technolc^ies  are  allowed,  priority  is  given  to  projects 
that  would  result  in  the  initiation  of  an  acquisition.  There  are  also  two  types  of 
procurement  testing:  qualification  and  comparative  testing.  Qualification  testing 
determines  if  a  prospective  system  meets  the  stated  mission  requirements;  while 
comparative  testing  performs  side-by-side  testing  on  multiple  items  that  are  poten¬ 
tially  capable  of  meeting  the  requirement.  If  domestic  items  are  potential  candi¬ 
dates  in  the  comparison  test,  the  program  sponsor  must  ensure  funds  are  available 
to  execute  that  portion  of  the  test  before  final  approval  of  the  program  as  an  FCT. 
Numerous  examples  of  programs  that  have  completed  the  FCT  process  are  found 
on  the  CTO  Web  site.  FCTs  have  led  to  more  than  $6.2  billion  in  procurements, 
saving  an  estimated  $4.4  billion  in  development  costs. 

Advance  Concept  Technology  Demonstration  (ACTD)  program  is  another  OSD 
program  managed  by  the  Deputy  Under  Secretary  of  Defense  for  Advanced  Sys¬ 
tems  and  Concepts  (DUSD(AS&C)).  The  purpose  of  an  ACTD  is  to  demonstrate 
the  effectiveness  of  mature  or  maturing  technology  to  meet  a  critical  user  need. 
The  potential  effectiveness,  maturity  of  their  respective  technologies,  and  the  user 
needs  being  met  are  key  criteria  against  which  candidate  programs  are  evaluated. 
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Candidate  technologies  with  a  high  d^ree  of  maturity  and  that  could  have  revolution¬ 
ary  impacts  on  operational  capabilities  have  an  excellent  chance  of  being  selected  and 
funded  under  this  program. 

Each  ACTD  is  executed  by  a  lead  agency  and  must  be  sponsored  by  a  user.  The 
Joint  Requirements  Oversight  Council  (JROC)  will  review  the  candidate  ACTDs  and 
make  recommendations  concerning  their  selection  and  execution.  Programs  selected 
will  nominally  receive  10  to  30  percent  of  the  ACTD’s  funding  from  OSD.  Each  ACTD 
will  be  executed  under  an  oversight  group,  chaired  by  DUSD(AS&C).  A  major  ob¬ 
jective  of  the  ACTD  program  is  to  transition  technologies  with  demonstrated  militaiy 
utility  seamlessly  into  acquisition  programs.  With  the  requirement  for  the  technology 
to  be  mature,  in  order  to  qualify  for  an  ACTD,  the  program  should  enter  the  traditional 
acquisition  cycle  late  in  the  continuum  and  reduce  the  time  to  field  the  system.  However, 
since  the  technology  is  often  demonstrated  on  a  prototype  system,  it  is  likely  to  need 
some  development  and  less  likely  than  an  FCT  to  proceed  directly  to  a  milestone  C 
decision.  Examples  and  instructions  to  initiate  an  ACTD  can  be  found  at  http:// 
vAvw.acq.osd.mil/actd.  Current  Air  Force  direction  is  under  revision  in  a  draft  (AFI  10- 
2302),  Advanced  Concept  Technology  Demonstration. 


The  All"  Tone  established  its  battlelab  program 
to  rapidly  identify  and  demonstrate  the 
military  utility  of  innovative  near-term 
iontepts  for  the  warfighter. 


The  Air  Force  estaHished  its  battlelab  program  to  rapidly  identify  and  demonstrate 
the  military  utility  of  innovative  near-term  concepts  for  the  warfighter.  The  Air  Force 
stood  the  program  up  with  the  signing  of  Air  Force  Policy  Directive  (AFPD)  10-19, 
Air  Force  Battlelab  Policy,  October  1,  1997.  This  established  the  initial  six  battlelabs: 
1)  Air  Expeditionaiy  Force  (AEF)  Battlelab,  2)  Command  and  Control  Battlelab  (C2B), 
3)  Force  Protection  Battlelab,  4)  Information  Warfare  Battlelab,  5)  Space  Battlelab, 
and  6)  Unmanned  Aerial  Vehicle  Battlelab.  A  seventh,  the  Air  Mobility  Battlelab  was 
added  later.  The  HQ  USAF  Deputy  Chief  of  Staff  (DCS)  for  Warfighting  Integration, 
AF  Battlelabs  Innovation  Division,  provides  overarching  battlelab  guidance,  policy, 
and  oversight.  AFPD  10-19  (1997)established  a  close  relationship  between  the 
battlelabs  and  both  Air  Force  Research  Laboratories  (AFRL)  and  Air  Force  Materiel 
Comtnand  (AFMC)  from  the  start.  AFRL  provides  the  battlelabs  with  technical 
expertise;  demonstration  facilities;  data  analysis;  demonstration  ideas;  and  full-time 
on-site  representation  at  the  battlelabs.  The  battlelabs  offer  AFRL  the  opportunity  to 
match  current  operational  needs  with  existing  research  efforts  in  battlelab  demon¬ 
strations.  AFMC  provides  transitional  support  to  both  the  battlelabs  and  AFRL,  to 
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further  expedite  fielding  of  successful  initiatives.  Battlelab  initiatives  are  funded  through 
both  procurement  and  operations  and  maintenance  (O&M)  funds.  Again,  the  program 
is  unlikely  to  proceed  directly  to  a  milestone  C  decision  based  on  testing  a  prototype 
system.  Links  to  all  battlelabs  may  be  found  on  the  Space  Battlelab  Web  site,  http;/ 
/www.schriever.af.mil/battlelab.  For  additional  information,  reference  AFI  10-2303. 


OTHER  ACQUISITION  TEST  TOOLS 

A  major  concern  of  the  Air  Force  research  laboratories  is  transitioning  basic  research 
into  field-able  systems.  As  a  technology  progresses  to  execution  under  6.3  research 
funds,  the  laboratory  is  expected  to  demonstrate  its  viability  in  an  operational  significant 
way.  This  demonstration  talffls  place  in  the  form  of  critical  experiments  and  advanced 
technology  demonstrations  (ATDs).  To  engender  advocacy  among  the  user  and  acqui¬ 
sition  communities,  laboratory  program  managers  must  encourage  their  participation 
in  the  planning  and  execution  of  these  tests.  Addressing  user  and  acquisition  concerns 
at  this  point  increases  the  likelihood  of  a  successful  transition  to  an  acquisition  pro¬ 
gram.  The  identification  of  show  stopper  issues  at  this  stage  will  also  save  the  Ah  Force 
valuable  research,  development,  test,  and  evaluation  (RDT&E)  dollars.  The  Air  Force 
Chief  of  Staff  has  also  established  applied  technology  councils  (ATCs)  to  assist  in 
identifying  funding  sources  for  technologies  that  successfully  complete  ATDs. 


TmtiHional  T&£  tools,  with  the  adaptations  made 
during  the  last  detade,  have  proven  flexible  enough 
to  address  many  of  the  requirements 
of  the  agile  arquisition  environment. 


In  addition  to  the  fore  mentioned  tests,  the  Air  Force  and  OSD  have  many  on¬ 
going  tests,  exercises,  and  experiments.  Each  of  these  offers  opportumties  to  identify 
new  needs  and  examine  potential  solutions.  Astute  program  managers  and  test  directors 
may  gain  access  to  resources  and  environments  that  would  otherwise  be  unavailable 
by  participating  in  these  tests.  Exploiting  them  will  often  also  save  both  time  and 
money,  as  well  as  address  test  requirements  that  may  be  otherwise  imtestable.  The 
following  are  a  few  of  the  major  efforts  and  their  primary  purpose: 

■  Joint  Test  and  Evaluation  (JT&E)-numerous  multi-year  tests  examining  inter-service 
integration  issues  in  designated  mission  areas, 

■  Joint  Expeditionary  Force  Exercise  (JEFXj-annuaEbiennial  exercises  focused  on 
CSAF-directed  integration  issues. 
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■  Joint  Interoperability  Testing  (JIT)-certifies  system  as  compliant  with  all  joint 
interoperability  requirements, 

■  Weapon  Systems  Evaluation  Program  (WSEP)-annual  weapons  delivery 
evaluations, 

•  Tactics  Development  and  Evaluation  (TD&E)-evaluations  of  specifically  tasked 
concept  of  operations  and  development  of  potential  alternatives,  and 

■  Foreign  Materiel  Exploitation  (FME)-testing  of  newly  acquired,  foreign  militaiy 
hardware. 


SUAAAAARY 

Traditional  T&E  tools,  with  the  adaptations  made  during  the  last  decade,  have  proven 
flexible  enough  to  address  many  of  the  requirements  of  the  agile  acquisition  environment. 
With  the  continued  drive  to  improve  the  acquisition  process,  the  T&E  process  must 
continue  to  evolve  and  address  future  acquisition  innovations.  The  incentive  evolution¬ 
ary  test  tools  bring  more  to  the  table  than  funding— visibility.  The  headquarters-level 
sponsorship  required  to  become  an  OSD-  or  Air  Force-level  test  program  can  carry 
effective  systems  into  acquisition  programs,  providing  the  political  clout  necessary  to 
find  the  required  fimding.  Both  the  traditional  and  evolutionary  test  tools  will  continue 
to  be  invaluable  tools  in  delivering  operational  capabilities  to  the  warfighter.  Additional 
insight  into  the  scope  and  use  of  these  tests  may  be  found  by  referring  to  referenced 
documents  and  Web  sites. 
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Center  and  the  Electronic  Depot  of  the  Air  Force,  where  he  has 
served  as  an  electronics  engineer,  operations  research  onalyst, 
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Knowledge  Sharing  Forum 


riie  Defense  Acquisition  University  is 
pleased  to  announce  its  first  Defense 
Acquisition  Review  Journal  (ARJ) 
knowledge  sharing  forum.  The  foru 
will  be  on  Performance- Based 
Acquisition  (PBA)  and  will  take 
at  the  DAU’s  Northeast  Campu 
one-day  event  will  be  held  Tu 
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\  \ 
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beryl. harman(ff'dau.Aj1i4}r  via  phone  at 


703.804.5403 


rd  to  seeing  you  there! 
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SUBMISSIONS 

We  welcome  submissions  from  anyone  involved  in  the  defense  acquisition  process. 
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must  accompany  your  submission.  Abstracts  give  readers  the  opportunity  to  quickly  review 
an  articles’  content  and  also  allow  information  services  to  index  and  retrieve  articles. 

The  introduction,  which  should  not  be  labeled,  opens  the  body  of  the  paper  and  states 
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■  Is  the  research  question  significant? 

■  Are  research  instruments  reliable  and  valid? 
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Government  Printing  Office,  1994,  Circular  92:  Copyright  Law  of  the  United  States  of 
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of  the  written  permission  to  the  Managing  Editor  before  puWication. 


COPYRIGHT  POLICY 
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■  The  author  cannot  obtain  official  permission  to  use  previously  copyrighted  material  in 
the  article. 
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■  The  author  will  not  allow  DAU  to  post  the  article  with  the  rest  of  the  ARJ  issue 
on  our  home  page. 

■  The  author  requires  that  unusual  copyright  notices  be  posted  with  the  article. 
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ing  charts  and  slides  to  a  look  similar  to  those  in  previous  issues  of  the  ARJ. 

The  author  (or  corresponding  author  in  cases  of  multiple  authorship)  should  attach 
to  the  manuscript  a  signed  cover  letter  that  provides  all  of  the  authors’  names,  mailing 
and  email  addresses,  telephone  and  fax  numbers.  The  letter  should  verify  that  the 
submission  is  an  original  product  of  the  author;  that  it  has  not  been  published  before; 
and  that  it  is  not  under  consideration  by  another  publication.  Details  about  the 
manuscript  should  also  be  included  in  this  letter:  for  example,  title,  word  length,  a 
description  of  the  computer  application  programs,  and  file  names  used  on  enclosed 
diskettes  or  in  email  attachments,  etc. 


AUTHOR  PHOTOS 

Please  send  us  a  cover  letter;  biographical  sketch  for  each  author  (not  to  exceed 
70  words);  head  and  shoulder  print(s)  or  digitized  photo(s)  (saved  at  300  pixels  per 
inch,  at  least  5X7  inches,  and  as  a  TIFF  file);  prints  of  photos  will  be  accepted  and 
returned  upon  request,  one  copy  of  the  printed  manuscript;  and  any  diskettes.  These 
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ilgj  GUIDELINES  FOR  CQNTRtBUTDRS 


DEFENSE  ARJ  THEME  AND  PRINT  SCHEDULE 

The  Defense  ARJ  is  published  in  quarter^  theme  editions.  Please  consult  the  DAU 
home  page  for  current  themes  being  solicited.  See  print  schedule  below. 

Due  Date  Publication  Date 

30  December  2004  -b-Aprib^OOr  t  ^ 

30  October  2004  ^ 

30  April  2005  1  August  2005 

30  August  2005  1  December  2005 

In  most  cases,  the  author  will  be  notified  that  the  submission  has  been  received 
within  48  hours  of  its  arrival.  Following  an  initial  review,  submissions  will  be  referred 
to  referees  and  for  subsequent  consideration  by  the  Executive  Editor,  Defense  ARJ, 

Contributors  m^  direct  their  questions  to  the  Managing  Editor,  Defense  ARJ,  at 
the  address  shown  above,  or  by  calling  (703)  805-3801  (fax:  (703)  805-2917),  or  via 
the  Internet  at  norene.t^lor@dau.mil. 

The  DAU  Home  Page  can  be  accessed  at:  http://vww.dau.mil. 
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